Process and apparatus for extracting and recognizing figure elements using division into receptive fields, polar transformation, application of one-dimensional filter, and correlation between plurality of images

ABSTRACT

An image processing process wherein an image is divided into small areas, a polar transformation is applied to the image in each of the small areas, and image processing is performed based on the result of the polar transformation. Further, each of a plurality of images is divided into small areas, a polar transformation is applied to the image in each of the small areas for each of the plurality of images, and correlation is obtained between the results of the polar transformation for the plurality of images. One-dimensional filter processing is applied to the polar-transformed output.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing process forextracting three-dimensional features of an object.

It is necessary to spatially measure a distance to an obstacle or anobject when a robot moves to avoid the obstacle or precisely manipulatethe object.

The range (50 degrees) of sight of the conventional robots is notenough, in particular, when the robots move in a narrow environment suchas in an industrial plant or in a warehouse. It is necessary to performthree-dimensional measurement recognizing the environmental conditionswith a field of view comparable to the range (180 degrees) of the humaneye. A fisheye lens is used for performing the measurement with a widefield of view. However, the images obtained through the fisheye lens aredistorted, and therefore it is difficult to precisely process thedistorted images, and a special image processing is required.

Since almost all the objects to be manipulated and the environmentalconditions in an industrial plant or in a warehouse are artificial, theyare constituted basically by straight lines and cylinders for ease inmanufacture. Therefore, the objects are imaged as straight lines on aninput screen. Thus, preprocessing of an image for precisely extracting aline segment is indispensable for movement or operations of the robots.The robots can perform operations such as avoiding an obstacle orapproaching an object by using the line segment as a clue.

The function for precisely extracting a line segment from an image of awide field of view (fisheye lens image) is indispensable for movement oroperations of the robots in a narrow environment such as in anindustrial plant or in a warehouse.

2. Description of the Related Art

2.1 The applicants have already invented and proposed athree-dimensional (stereoscopic) moving view (see for example, theJapanese Examined Patent No. 3-52106, or Kawakami in Kagaku Asahi, June1987). In the kinetic stereopsis, a three-dimensional (stereoscopic)perception is obtained based on the motion parallax caused by movement,and the image obtained when moving a fisheye lens camera is processed ona sphere to perform three-dimensional (stereoscopic) measurement of aline segment, a point, a cylinder, and the like. Thus, a line segmentcan be three-dimensionally (stereoscopically) measured over the range ofhuman sight (180 degrees).

FIGS. 1A and 1B are diagrams for explaining a spherical mapping. Theimage input through a fisheye lens is equivalent to an image obtained bythe projection on a sphere, and is distorted. Therefore, an operationcalled a spherical mapping (which is precisely denoted as a polartransformation or a dual transformation on a sphere) is required. Thespherical mapping is an operation wherein an arbitrary point P on asphere is transformed to a great circle R (a largest circle on a spherecorresponding to an equator) having a pole thereof at the point P, asindicated in FIG. 1A. When drawing great circles R1, R2, R3, . . .respectively having their poles at points P1', P2', P3', . . . , wherethe points P1', P2', P3', . . . are respectively obtained by projectingpoints P1, P2, P3 onto the sphere, . . . which constitute a line segmentL, the great circles necessarily cross at a point S, as indicated inFIG. 1B. The intersecting point S is a characteristic point having aone-to-one correspondence to the line segment L. The longer the linesegment L is, the larger the number of the points in the line segment L,the larger the number of the great circles, and therefore the higher thedegree of superimposition of the great circles at the point S. Thus, aline segment is extracted as a point corresponding to the line segmenton a sphere, and the length of the line segment can be measured byobtaining a histogram of the degree of superimposition of the greatcircles at respective points. The point S corresponding to the linesegment L can be expressed in geometry as "a pole having as a polar line(great circle) a projection L' of a line segment L onto a sphere".

FIG. 2 is a diagram illustrating the construction of a system forperforming three-dimensional (stereoscopic) measurement by a kineticstereopsis using a spherical mapping. When an image IMG is input from afisheye lens built in the spherical camera 1, the contour extractingportion 2 extracts a contour, compresses information, and writes theinformation in a spherical mapping image memory 2a built in the contourextracting portion 2. The contour of an object can be extracted bydetecting with differentiation points where the brightness in the imageis maximized.

Next, the line segment extracting portion 3 extracts a line segment onthe sphere (hereinafter called a great circle) by concentrating the linesegment into a point in the spherical mapping process. The process forextracting a line segment is an important process in thethree-dimensional (stereoscopic) measurement system, and the majorportion of processing time is spent for the process. For example, toextract the line segment, the polar transformation portion 3a transformseach contour to a great circle by the spherical mapping, and writesinformation on each great circle into the mapping memory 3b. The addressof the mapping memory 3b is given by the longitude α and latitude βindicating a point P on a sphere CB as indicated in FIG. 3. Each celldesignated by the address in the mapping memory 3b is constituted by,for example, a counter, and the counter is incremented by one every timea writing operation is carried out. After all the contour points aretransformed to great circles by the spherical mapping, the S pointdetection portion 3c scans the respective cells in the mapping memory3b, and obtains a peak position of the count values. The peak positionis the pole (point S) of the line segment as explained with reference toFIG. 1B. Thus, a line segment is extracted.

Based on the "point" data extracted by the concentration of a linesegment as explained above, the three-dimensional measurement thereafteris easily carried out (cf., for example, the international laid-openWO90/16037 by Morita et al.). When the "point" data is input into theline segment measurement portion 4, the three-dimensional data(orientation and distance) of a straight line portion in a screen isoutput. For example, the spherical camera 1 is moved in a straightdirection when measuring an orientation of a line segment. Theabove-mentioned operation of extracting a line segment is repeated forthe successively obtained images. FIG. 4 is a diagram illustratingrelative locations of the line segment and the camera when the sphericalcamera 1 is moved in a straight direction. The poles S, S', S", . . . onthe sphere, corresponding to the relatively moved line segment L, L', L". . . , line up on a great circle. When performing the spherical mappingfor the respective points, and drawing great circles R, R', R", . . . ,the intersecting point Ss of the great circles is a pole of a greatcircle on which the poles S, S', S", . . . lie. The vector directed fromthe center of the sphere to the point Ss is parallel to the actual linesegment L. Thus, an orientation of the line segment is obtained.Geometrically, the point Ss is a point generated by projecting avanishing point at infinity on the line segment L onto the sphere.Namely, the orientation of the line segment is determined based ontheory of the perspective projection method. The extracted poles S, S',S", . . . are respectively transformed to great circles by the sphericalmapping, and the information on the great circles are written in themapping memory. The mapping memory is scanned to obtain a peak positionof the count values as a vector of an orientation of the line segment.As understood from FIG. 4, the point Ss corresponds to a group ofparallel lines

The principle of measuring a depth to the object is explained below.First, a process of measuring a distance from a camera to a point P on atwo-dimensional plane is explained with reference to FIG. 5A. When thecamera is moved as C⁰, C¹ C², . . . , the direction in which the point Pis viewed varies as Sk⁰, Sk¹, Sk², . . . . When drawing straight linesin the directions in respective timings, these lines cross at a point P.Therefore, when the intersecting point P is obtained, the distance fromthe initial position C⁰ of the camera to the point P of the object isgiven by the length of the line segment C⁰ P. An operation similar tothe above is carried out on a sphere. In FIG. 5B, the plane on the rightside corresponds to FIG. 5A, and placed in perpendicular to the linesegment OΣ. The correspondence between the plane and the sphere isindicated by dashed lines, and the direction of the movement of thecamera is indicated by V. It is assumed that the point P is viewed asP⁰, P¹, P², . . . , and is projected on the sphere as Sk⁰, Sk¹, Sk² whenthe camera is moved by a pitch Δx⁰. The point Σ is a pole of the greatcircle R generated by mapping the trace of the point P, and is obtainedas an intersecting point of a group of great circles obtained from thepoints Sk⁰, Sk¹, Sk² by the spherical mapping. A time axis (τ-axis) isassumed on a quarter circle from the point Σ to the end point v of thevector V on a circle R' passing through the points v and Σ, and thepoint Σ is assumed to correspond to τ=0, i.e., C⁰. The points thelengths (expressed by angles) on the sphere to which from the point C⁰are equal to τ=arctan (iη) (i=1, 2, . . . ) are denoted by C¹, C², . . ., where η=Δx0/R0. This operation means to plot the points C⁰, C¹, C², .. . with a pitch of 1/R0 on the plane on the right side in FIG. 5B. Asunderstood by making i→∞ in the above equation of τ, the end point vcorresponds to a point at infinity.

Next, considering that a straight line on a plane corresponds to a greatcircle on a sphere, the point C⁰ and the point Sk⁰, the point C¹ and thepoint Sk¹, the point C² and the point Sk², . . . , are connected bygreat circles, respectively. The great circles thus obtained cross at apoint Q. Thus, the distance from the initial position C⁰ of the camerato the point P is given by the product of R0 and a tangent of the lengthof the arc C⁰ Q, where lengths on the sphere are expressed by angles.

Next, the above "point" data is input into the cylinder measurementportion 5, the three-dimensional data (an orientation, a distance, and adiameter) is output therefrom. As explained before, parallel lines areobtained, the cylinder and the diameter thereof can be obtained from theparallel lines, and the orientation and the distance are also obtainedin the same way as the case of the line segment.

Although almost all of the environmental conditions can be measured bythe straight lines and the cylinders as above in the artificialenvironment such as an industrial plant, raw "point" data may be inputinto the point measurement portion 6 (shown in FIG. 2) to performthree-dimensional measurement the location of each point in the spacewhen it is required to recognize environmental conditions other than theabove.

2.2 The above three-dimensional measurement system contains thefollowing problems. One of the problems is to increase the speed of theoperation and to reduce the size of the system, and the other is tosuppress interference.

As explained above, the major portion of the processing time is spentfor the process of extracting a line segment in the three-dimensionalmeasurement using the fisheye lens. The major reason is that each pointof an input image is transformed by the spherical mapping to extract theline segment, i.e., each point in the input image is transformed to agreat circle on a sphere increasing the dimension. When the size of theinput image is assumed to be N×N, the spherical mapping is required totransform each point to a great circle having a length N. The amount ofprocessing to N³, which is N times the data amount N² in the inputimage, is required to transform each point to a great circle having alength N, and this makes increasing the speed of the operationdifficult. Although parallel provision of hardware may increase thespeed, this increases the hardware size. Thus, both the increase inspeed and reduction of hardware size are required at the same time.

In the case where another great circle exists in an orientation near thedirection of the great circle corresponding to the object undermeasurement, in the operation of extracting a line segment by thespherical mapping, the accuracy of the extracted line segment isdeteriorated due to interference of the two great circles. For example,in the case where ridge-lines AL and BL of solid bodies A and B cross atan angle near 180° as indicated in FIG. 6, these ridge-lines interferewith each other, and are detected as an obscure line. Namely, preciseline detection is impossible. To perform precise measurement of anobject in a complicated environment, the extraction of a line segmentwith suppressed interference is required together with the increase inspeed and reduction of size.

2.3 It is necessary to recognize three-dimensional conditions of theenvironment of a robot when the robot moves or controls an automaticoperation thereof. There are two methods for recognizingthree-dimensional conditions of the environment. One method is the"binocular stereopsis" whereby the depth is measured in accordance withthe principle of trigonometrical survey using the binocular parallaxbetween the right and left eyes, and the other is the "kineticstereopsis" whereby the three-dimensional (stereoscopic) perception isobtained using the motion parallax generated by moving of a viewer. The"binocular stereopsis" has been developed for years. Although it isnecessary to extract corresponding portions in the images obtained byright and left eyes, the extraction of the corresponding portions isdifficult by the conventional technique.

FIG. 7 is a diagram for explaining the principle of the "binocularstereopsis". In FIG. 7, it is assumed that objects are placed at the twopoints A and B on a plane. Although only the directions toward theobjects A and B from the two eyes can be recognized, respectively, thedepth to each of the objects A and B is recognized as an intersectingpoint at which the directions of the two eyes cross. Namely, asindicated in FIG. 8, the depth D is obtained by the following equation,

    D=d/(tan ρ.sub.L +tan ρ.sub.R)

where d denotes a distance between two eyes, ρ_(L) and ρ_(R) denoteangles between the direction perpendicular to the line on which the twoeyes lie, and the lines of sight by the left and right eyes,respectively.

However, the directions of the two eyes cross at other points. Namely,the direction of the left eye seeing the object A crosses with thedirection of the right eye seeing the object B at the point β. The pointβ is an untrue (false) point. Similarly, an untrue point α may begenerated. These untrue points must be eliminated in the "binocularstereopsis".

Since the capability of recognizing a shape of an object is developed ina human cerebrum, a human being can easily eliminate the untrue points.However, in the conventional technique of the "binocular stereopsis", itis difficult to precisely recognize the corresponding points, andtherefore the development of the technique of precisely recognizing thecorresponding points is required.

SUMMARY OF THE INVENTION

A first object of the present invention is to provide an imageprocessing process and apparatus wherein the processing speed isincreased and the size thereof can be reduced.

A second object of the present invention is to provide an imageprocessing process and apparatus whereby a line segment can be preciselyextracted without interference with another line segment.

A third object of the present invention is to provide an imageprocessing process and apparatus wherein the amount of filter processingcan be reduced, thereby filtering with a large mask is possible, and aline segment can be precisely extracted from an obscure image.

A fourth object of the present invention is to provide an imageprocessing process and apparatus whereby an edge can be extracted, and aline and a gap can be prominently extracted.

A fifth object of the present invention is to provide a correlationprocessing process and apparatus wherein the corresponding points in aplurality of images can be determined with a simple procedure and asmall amount of processing, and the function of the binocular stereopsiscan be realized.

A sixth object of the present invention is to provide an imageprocessing process and apparatus whereby a moving object can be tracedbased on correspondence between images at different times, and thedirection and the velocity of the movement can be measured.

A seventh object of the present invention is to provide a correlationprocessing process and apparatus which can be applied to a textureanalysis for examining a degree of similarity of a pattern.

An eighth object of the present invention is to provide a correlationprocessing process and apparatus whereby the binocular stereopsis, thepursuit of a moving object, and the texture analysis can be certainlyperformed by carrying out color correlation by using three primarycolors, three essential color elements, or the like.

A ninth object of the present invention is to provide a correlationprocessing process and apparatus whereby corresponding tangential lines(a line, a gap, and an edge) in a plurality of figures can be obtained,and the location, the orientation, the parallax, and the velocity can bequantitatively obtained.

A tenth object of the present invention is to provide a correlationprocessing process and apparatus wherein precise filtering can beperformed, and the process and apparatus can be applied to the"extraction of a feature which is seen as the same by the left and righteyes", the "pursuit of the same feature as the preceding image", and thelike.

An eleventh object of the present invention is to provide a correlationprocessing process and apparatus whereby the binocular stereopsis andthe pursuit of an object having an obscure contour are carried out byusing gradual variations of brightness and hue, and the like as a clue.

According to the first aspect of the present invention, there isprovided an image processing process containing: a first step fordividing an original image into small areas; a second step for applyingpolar transformation to the original image in each of the small areas;and a third step for performing image processing on the result of thepolar transformation.

Stated in more detail, there is provided an image processing processcontaining: a first step for dividing an original image into smallareas; a second step for obtaining a curve on a predetermined dual planecorresponding to each pixel in the original image in each of the smallareas; a third step for obtaining a polar-transformed image on the dualplane by accumulatively storing in a memory having a storage area foreach pixel on the dual plane, a value of each pixel in the originalimage, as a value of each pixel through which the curve obtained by thepolar transformation passes on the dual plane; and a fourth step forperforming image processing on the polar-transformed image on the dualplane for each of the small areas.

Stating more in detail, there is provided an image processing processcontaining: a first step for dividing an original image into smallareas; a second step for obtaining a curve on a predetermined dual planecorresponding to each pixel in the original image in each of the smallareas; a third step for obtaining a polar-transformed image on the dualplane by accumulatively storing in a memory having a storage area foreach pixel on the dual plane, a value of each pixel in the originalimage, as a value of each pixel through which the curve obtained by thepolar transformation passes on the dual plane; and a fourth step forperforming image processing on the polar-transformed image on the dualplane for each of the small areas.

According to the second aspect of the present invention, there isprovided an image processing process containing: a first step fordividing an original image into small areas; a second step for applyingpolar transformation to the original image in each of the small areas; athird step for applying one-dimensional filtering to the result of thepolar transformation; and a fourth step for performing image processingon the result of the third step.

Stated in more detail, there is provided an image processing processcontaining: a first step for dividing an original image into smallareas; a second step for obtaining by a predetermined polartransformation a curve on a predetermined dual plane corresponding toeach pixel in the original image in each of the small areas; a thirdstep for obtaining a polar-transformed image on the dual plane byaccumulatively storing in a memory having a storage area for each pixelon the dual plane, a value of each pixel in the original image, as avalue of each pixel through which the curve obtained by the polartransformation passes on the dual plane; a fourth step for applyingone-dimensional filtering on the polar-transformed images for each ofthe small areas; and a fifth step for performing image processing on thedual plane in which the one-dimensional filtering has been applied.

Stated in more detail, there is provided an image processing processcontaining: a first step for dividing each of a plurality of originalimages into small areas; a second step for obtaining a curve on apredetermined dual plane corresponding to each pixel in the originalimage in each of the small areas, for each of a plurality of originalimages; a third step for obtaining a polar-transformed image on the dualplane by accumulatively storing in a memory having a storage area foreach pixel on the dual plane, a value of each pixel in the originalimage, as a value of each pixel through which the curve obtained by thepolar transformation passes on the dual plane; and a fourth step forobtaining correlation between the polar-transformed images obtained inthe third step for the plurality of original images.

According to the third aspect of the present invention, there isprovided an image processing process containing: a first step fordividing each of a plurality of original images into small areas; asecond step for applying polar transformation to the original image ineach of the small areas, for each of the plurality of original images;and a third step for obtaining correlation between the results of thepolar transformation obtained in the second step for the plurality oforiginal images.

Stated in more detail, there is provided an image processing processcontaining: a first step for dividing each of a plurality of originalimages into small areas; a second step for obtaining a curve on apredetermined dual plane corresponding to each pixel in the originalimage in each of the small areas, for each of a plurality of originalimages; a third step for obtaining a polar-transformed image on the dualplane by accumulatively storing in a memory having a storage area foreach pixel on the dual plane, a value of each pixel in the originalimage, as a value of each pixel through which the curve obtained by thepolar transformation passes on the dual plane; and a fourth step forobtaining correlation between the polar-transformed images obtained inthe third step for the plurality of original images.

Further, the third aspect of the present invention includes thefollowing aspects.

(1) An image processing process containing: a first step for dividingeach of first and second original images into small areas; a second stepfor obtaining first and second polar-transformed images on the originalimages by applying polar transformation to each of the small areas ineach of the first and second original images; a third step for obtaininga shifted image by shifting one or more coordinates among a plurality ofcoordinates determining a location of each pixel on the dual plane forone of the first and second polar-transformed images; a fourth step forobtaining a correlation value between the other of the first and secondpolar-transformed images and the shifted image, varying shift value(s)of the one or more coordinates within predetermined range(s),respectively; and a fifth step for obtaining shift value(s) and theplurality of coordinates which maximize the correlation value.

(2) An image processing process containing: a first step for dividingeach of first and second original images into small areas, where thefirst original image is shot at a first time, and the second originalimage is shot at a second time; a second step for obtaining first andsecond polar-transformed images on the original images by applying polartransformation to each of the small areas in each of the first andsecond original images; a third step for obtaining a shifted image byshifting one or more coordinates among a plurality of coordinatesdetermining a location of each pixel on the dual plane for one of thefirst and second polar-transformed images; a fourth step for obtaining acorrelation value between the other of the first and secondpolar-transformed images and the shifted image, varying shift value(s)of the one or more coordinates within predetermined range(s),respectively; and a fifth step for obtaining shift value(s) and theplurality of coordinates which maximize the correlation value.

(3) An image processing process containing: a first step for dividing anoriginal image into a plurality of small areas; a second step forobtaining a polar-transformed image by applying polar transformation tothe original image in each of the small areas; a third step forobtaining a shifted image by shifting one or more coordinates among aplurality of coordinates determining a location of each pixel on thedual plane for the polar-transformed image for a first small area amongthe plurality of small areas; a fourth step for obtaining a correlationvalue between the polar-transformed image obtained in the second stepfor a second small area and the shifted image obtained in the third stepfor the first small area, varying shift value(s) of the one or morecoordinates within predetermined range(s), respectively; a fifth stepfor obtaining shift value(s) and the plurality of coordinates whichmaximize the correlation value; and a sixth step for performing theoperations of the third to fifth steps, varying the combination of thefirst and second small areas.

(4) An image processing process containing: a first step for separatingfirst and second color original images into first, second, and thirdoriginal images respectively comprised of intensity distributions ofthree elements determining a color image; a second step for dividingeach of first, second, and third original images into small areas; asecond step for obtaining a polar-transformed image on the originalimages by applying polar transformation to each of the small areas ineach of first, second, and third original images; a third step forobtaining a shifted image by shifting one or more coordinates among aplurality of coordinates determining a location of each pixel on thedual plane for one of the first and second polar-transformed images; afourth step for obtaining a correlation value between the other of thefirst and second polar-transformed images and the shifted image, varyingshift value(s) of the one or more coordinates within predeterminedrange(s), respectively; and a fifth step for obtaining shift value(s)and the plurality of coordinates which maximize the correlation value.

(5) An image processing process containing: a first step for separatingfirst and second color original images into first, second, and thirdoriginal images respectively comprised of intensity distributions ofthree elements determining a color image; a second step for dividingeach of first, second, and third original images into small areas; asecond step for obtaining a polar-transformed image on the originalimages by applying polar transformation to each of the small areas ineach of first, second, and third original images; a third step forobtaining a shifted image by shifting one or more coordinates among aplurality of coordinates determining a location of each pixel on thedual plane for one of the first and second polar-transformed images; afourth step for obtaining a correlation value between the other of thefirst and second polar-transformed images and the shifted image, varyingshift value(s) of the one or more coordinates within predeterminedrange(s), respectively; and a fifth step for obtaining shift value(s)and the plurality of coordinates which maximize the correlation value.

(6) An image processing process containing: a first step for dividing anoriginal image into a plurality of small areas; a second step forobtaining a polar-transformed image by applying polar transformation tothe original image in each of the small areas; a third step forobtaining a shifted image by shifting one or more coordinates among aplurality of coordinates determining a location of each pixel on thedual plane for the polar-transformed image for a first small area amongthe plurality of small areas; a fourth step for obtaining a correlationvalue between the polar-transformed image obtained in the second stepfor a second small area and the shifted image obtained in the third stepfor the first small area, varying shift value(s) of the one or morecoordinates within predetermined range(s), respectively; a fifth stepfor obtaining a summed correlation value by summing the correlationvalue for the shift value(s) and the one or more of the plurality ofcoordinates other than at least one of the shift value(s) and the one ormore of the plurality of coordinates; and a sixth step for obtaining theat least one of the shift value and the plurality of coordinates whichmaximize the summed correlation value

(7) An image processing process containing: a first step for dividinginto small areas each of first and second original images among aplurality of original images which are respectively made by shooting anobject at a plurality of times, where the first original image is shotat a first time, and the second original image is shot at a second time;a second step for obtaining first and second polar-transformed images onthe original images by applying polar transformation to each of thesmall areas in each of the first and second original images; a thirdstep for obtaining a shifted image by shifting one or more coordinatesamong a plurality of coordinates determining a location of each pixel onthe dual plane for one of the first and second polar-transformed images;a fourth step for obtaining a correlation value between the other of thefirst and second polar-transformed images and the shifted image, varyingshift value(s) of the one or more coordinates and the time differencebetween the first and second times within predetermined ranges,respectively; and a fifth step for obtaining a summed correlation valueby summing the correlation value for the shift value(s) and the one ormore of the plurality of coordinates other than at least one of theshift value(s), the time difference, and the one or more of theplurality of coordinates; and a sixth step for obtaining the at leastone of the shift value, the time difference, and the plurality ofcoordinates which maximize the summed correlation value.

(8) An image processing process containing: a first step for dividing anoriginal image into a plurality of small areas; a second step forobtaining a polar-transformed image by applying polar transformation tothe original image in each of the small areas; a third step forobtaining a shifted image by shifting one or more coordinates among aplurality of coordinates determining a location of each pixel on thedual plane for the polar-transformed image for a first small area amongthe plurality of small areas; a fourth step for obtaining a correlationvalue between the polar-transformed image and the shifted image in aparameter space, varying shift value(s) of the one or more coordinateswithin predetermined range(s), respectively, where the shift value(s)and the plurality of coordinates determining the location are parametersin the parameter space; a fifth step for obtaining a summed correlationvalue by summing the correlation value for a predetermined point in theparameter space; and a sixth step for obtaining the at least one of theshift value and the plurality of coordinates which maximize the summedcorrelation value, where the at least one of the shift value and theplurality of coordinates correspond to the point in the parameter space.

(9) An image processing process containing: a first step for dividinginto small areas each of first and second original images among aplurality of original images which are respectively made by shooting anobject at a plurality of times, where the first original image is shotat a first time, and the second original image is shot at a second time;a second step for obtaining first and second polar-transformed images onthe original images by applying polar transformation to each of thesmall areas in each of the first and second original images; a thirdstep for obtaining a shifted image by shifting one or more coordinatesamong a plurality of coordinates determining a location of each pixel onthe dual plane for one of the first and second polar-transformed images;a fourth step for obtaining a correlation value between the other of thefirst and second polar-transformed images and the shifted image in aparameter space, varying shift value(s) of the one or more coordinatesand the time difference between the first and second times withinpredetermined ranges, respectively, where the shift value(s), the timedifference, and the plurality of coordinates determining the locationare parameters in the parameter space; and a fifth step for obtaining asummed correlation value by summing the correlation value for apredetermined point in the parameter space; and a sixth step forobtaining the at least one of the shift value, the time difference, andthe plurality of coordinates which maximize the summed correlationvalue, where the at least one of the shift value, the time difference,and the plurality of coordinates correspond to the point in theparameter space.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings:

FIGS. 1A and 1B are diagrams for explaining a spherical mapping (polartransformation on a sphere);

FIG. 2 is a diagram illustrating a three-dimensional measurement system;

FIG. 3 is a diagram for explaining an address in the mapping memory;

FIG. 4 is a diagram for explaining a three-dimensional orientation of aline segment;

FIGS. 5A and 5B are diagrams for explaining a depth to an objectivepoint;

FIG. 6 is a diagram for explaining interference with a background;

FIG. 7 is a diagram for explaining a problem in the binocularstereopsis;

FIG. 8 is a diagram for explaining a calculation of a depth in thebinocular stereopsis;

FIG. 9 is a diagram illustrating the basic construction of the firstaspect of the present invention;

FIG. 10 is a diagram for explaining a division into receptive fields;

FIGS. 11A and 11B are diagrams for explaining visual informationprocessing by a mammalian and a simulation model thereof;

FIGS. 12A, 12B, and 12C are diagrams for explaining a receptive fieldand a polar transformation in a hypercolumn;

FIG. 13 is a diagram indicating an original image used in a simulationbased on the model of the visual cortex of a mammal;

FIGS. 14A and 14B are diagrams for explaining the result of thesimulation (an example of a response in a receptive field);

FIG. 15 is a diagram indicating a regenerated image obtained by thesimulation;

FIG. 16 is a diagram for explaining a polar transformation on a sphere;

FIG. 17 is a diagram for explaining a polar transformation on acylinder;

FIG. 18 is a diagram for explaining a polar transformation on a plane;

FIG. 19 is a diagram for explaining a "plane projection & a polartransformation on a plane";

FIG. 20 is a diagram for explaining a "plane projection & a polartransformation on a cylinder";

FIG. 21 is a diagram for explaining a "plane projection & a polartransformation on a sphere";

FIG. 22 is a diagram for explaining a "cylinder projection & a polartransformation on a sphere";

FIG. 23 is a diagram for explaining a "sphere projection & a polartransformation on a plane";

FIG. 24 is a diagram for explaining a "cylinder projection & a polartransformation on a sphere";

FIGS. 25A and 25B are diagrams for explaining a result of a simulation(an example of a response for element receptive field response);

FIG. 26 is a diagram for explaining a result of a simulation(regenerated image);

FIG. 27 is a diagram for illustrating the basic construction of thesecond aspect of the present invention;

FIGS. 28A, 28B, 28C, 28D, 28E, 28F, and 28G are diagrams for explaininga principle of a one-dimensional filter according to the second aspectof the present invention;

FIGS. 29A, 29B, 29C, and 29D are diagrams indicating a result of asimulation by a two-dimensional convolution+the receptive field method;

FIGS. 30A, 30B, 30C, and 30D are diagrams indicating a result of asimulation by the receptive field method+a one-dimensional convolution;

FIGS. 31A, 31B, 31C, and 31D are diagrams illustrating variousprocessing methods;

FIG. 32 is a diagram illustrating an embodiment of the second aspect ofthe present invention;

FIG. 33 is a diagram illustrating a construction of the polartransformation circuit;

FIG. 34 is a diagram illustrating a one-dimensional filter circuit;

FIGS. 35A and 35B are diagrams for explaining the storing operation inthe one-dimensional filter memory;

FIGS. 36A, 36B, 36C, 36D, 36E, and 36F are diagrams for explainingresponses to an edge by even and odd function filters;

FIGS. 37A, 37B, 37C, and 37D are diagrams indicating the results ofsimulations by the two-dimensional second differential filters+the polartransformation+the one-dimensional differential filter;

FIGS. 38A and 38B are diagrams indicating a result of a simulation bythe conventional two-dimensional second differential filter+the polartransformation;

FIGS. 39A and 39B are diagrams illustrating the characteristic of askeleton filter;

FIG. 40 is a diagram indicating a one-dimensional filter using askeleton filter;

FIGS. 41A and 41B are diagrams illustrating the result of extraction ofan edge by a construction using a skeleton filter;

FIGS. 41C and 41D are cross-sectional views of the contouring maps ofFIGS. 41A and 41B, respectively;

FIG. 42 is a diagram illustrating a construction using a multi-filter;

FIG. 43 is a diagram for explaining interference;

FIG. 44 is a diagram illustrating a construction of apositive-negative-separation-type one-dimensional multi-stage filter;

FIG. 45 is a diagram for explaining improvement in the interference;

FIG. 46 is a diagram illustrating a construction of apositive-negative-separation-type one-dimensional multi-filter;

FIGS. 47A and 47B are diagrams for explaining a result of a simulation;

FIG. 48 is a diagram for explaining a regenerated image obtained by asimulation;

FIGS. 49A, 49B, and 49C are diagrams for explaining symbols representingfilters;

FIGS. 50A, 50B, and 50C are diagrams for illustrating the constructionsof various line extraction filters;

FIGS. 51A and 51B are diagrams for indicating a result of a simulationof line extraction by the receptive field division+the polartransformation;

FIGS. 52A, 52B, 52C, and 52D are diagrams for illustrating theconstructions of various edge extraction filters;

FIGS. 53A and 53B are diagrams indicating a result of a simulation ofedge extraction by the receptive field division+polartransformation+one-dimensional gr-filter;

FIGS. 54A and 54B are diagrams for indicating a result of a simulationof edge extraction by the receptive field division+polartransformation+one-dimensional gas-filter;

FIG. 55 is a diagram illustrating a first construction of a filteroutputting an edge by a positive signal;

FIG. 56 is a diagram illustrating a second construction of a filteroutputting an edge by a positive signal;

FIGS. 57A, 57B, and 57C are diagrams for illustrating the constructionsof various gap extraction filters;

FIG. 58 is a diagram illustrating a construction of a line-onlyextraction filter;

FIG. 59 is a diagram illustrating a construction of a line-edgeextraction filter;

FIG. 60 is a diagram illustrating a concrete construction of a line-edgeextraction filter;

FIG. 61 is a diagram illustrating a construction of apositive-negative-separation-type multi-filter for extracting a line;

FIGS. 62A and 62B are diagrams indicating a result of a simulation ofline extraction by the positive-negative-separation-type multi-filter;

FIG. 63 is a diagram illustrating a multi-filter for extracting an edge;

FIG. 64 is a diagram illustrating a construction of apositive-negative-separation-type multi-filter for extracting an edge;

FIGS. 65A and 65B are diagrams indicating a result of a simulation ofedge extraction by the positive-negative-separation-type multi-filter;

FIGS. 66A and 66B are diagrams for explaining a multi-filter having aconstruction comprised of a two-dimensional gas-filter+one-dimensionalgr-filter;

FIGS. 67A and 67B are diagrams for explaining examples of variations ofthe multi-filter;

FIG. 68 is a diagram for illustrating the basic construction of thethird aspect of the present invention;

FIG. 69 is diagrams for explaining the principle of the binocularstereopsis;

FIG. 70 is a diagram illustrating the construction of the embodiment ofthe third aspect of the present invention;

FIG. 71 is a flowchart of correlation processing;

FIGS. 72A and 72B are first diagrams for explaining a result of asimulation;

FIGS. 73A and 73B are second diagrams for explaining a result of asimulation;

FIG. 74 is a third diagram for explaining a result of a simulation;

FIG. 75 is a basic block diagram of correlation filtering;

FIGS. 76A and 76B are block diagrams of correlation filtering betweenspatially different images;

FIG. 77 is a block diagram of correlation filtering between imagesdifferent in time;

FIG. 78 is a block diagram of correlation filtering between imagesdifferent in time and space;

FIG. 79 is a block diagram illustrating the correlation filteringbetween receptive fields in the same image;

FIG. 80 is a block diagram illustrating the correlation filtering in thesame receptive field in the same image;

FIG. 81 is a block diagram illustrating the correlation filteringbetween different color images (three primary colors);

FIG. 82 is a block diagram illustrating the correlation filteringbetween different color images (three color elements);

FIG. 83 is a block diagram illustrating the correlation filteringbetween different color images (color difference signals);

FIG. 84 is a block diagram of ρ-axis correlation filtering for the sameθ;

FIG. 85 is a block diagram of ρ-axis correlation filtering for differentθ's;

FIGS. 86A, 86B, 86C, and 86D are diagrams for explaining correlation inthe θ-direction;

FIG. 87 is a block diagram of θ-axis correlation filtering for the sameρ;

FIG. 88 is a block diagram of correlation filtering in the ρ-θ plane;

FIG. 89 is a diagram for explaining the correlation parameter space;

FIG. 90 is a block diagram illustrating a total filter processingwherein a correlation parameter is projected;

FIG. 91 is a flowchart of a process wherein a correlation parameter isprojected in the σ-direction;

FIG. 92 is a diagram illustrating the construction of a natural filter;

FIG. 93 is a flowchart of the difference-type correlation processexplaining an address in the mapping memory;

FIGS. 94A and 94B are first diagrams indicating a result of a simulation(the three-dimensional view of an edge by two eyes);

FIGS. 95A and 95B are second diagrams indicating a result of asimulation (the three-dimensional view of an edge by two eyes);

FIG. 96 is a diagram for explaining a moving direction and a truevelocity vector of a tangential line;

FIG. 97 is a block diagram for measuring a moving direction andvelocity;

FIGS. 98A and 98B are diagrams for explaining detection of a movingdirection and velocity of a corner;

FIG. 99 is a diagram for explaining a response of C_(PRJ) -ρ(θ,τ) to apolygon;

FIGS. 100A and 100B are diagrams for explaining extraction of a sinewave by the inverse polar transformation;

FIG. 101 is a block diagram for measuring a moving direction andvelocity;

FIGS. 102A and 102B are diagrams for explaining a polar transformationfrom a random dot image;

FIGS. 103A, 103B, 103C, and 103D are diagrams for explaining a result ofa simulation in measurement of a moving direction and velocity of arandom dot image;

FIGS. 104A and 104B are diagrams for explaining a result of a simulationin measurement of a moving direction and velocity of a random dot image;

FIGS. 105A, 105B, 105C, 105D, and 105E are diagrams for explaining aresult of a simulation (moving direction and velocity of a line);

FIGS. 106A, 106B, 106C, 106D, and 106E are diagrams for explaining aresult of a simulation (moving direction and velocity of an edge);

FIG. 107 is a diagram for explaining a relationship between an offset ofa line and a binocular parallax;

FIGS. 108A, 108B, and 108C are diagrams for explaining thethree-dimensional view of an edge of an arbitrary figure by two eyes;

FIG. 109 is a diagram for explaining a method for calculating a distanceby the binocular stereopsis;

FIG. 110 is a block diagram of calculation of a parallax and a distanceby the binocular stereopsis;

FIGS. 111A, 111B, 111C, and 111D are diagrams for explaining a result ofa simulation by the three-dimensional view of a random dot image by twoeyes;

FIGS. 112A, 112B, and 112C are diagrams for explaining a result of asimulation by the three-dimensional view of a random dot image by twoeyes;

FIG. 113 is a block diagram of calculation of depth to a straight lineby the kinetic stereopsis;

FIG. 114 is a block diagram of calculation of depth to an arbitraryfigure by the kinetic stereopsis;

FIG. 115 is a block diagram in a generalized case;

FIGS. 116A and 116B are diagrams for explaining detection of a circle;and

FIGS. 117A, 117B, 117C, and 117D are diagrams for explaining a result ofa simulation of the detection of a circle.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

1. First Aspect of Present Invention

1.1 Basic Construction of First Aspect of Present Invention

FIG. 9 is a diagram illustrating the basic construction of the firstaspect (the receptive field method) of the present invention.

In FIG. 9, reference numeral 112 denotes an input memory for storing animage (input image) IMG of an object of a size equal to N×N, where theimage is projected on a predetermined input plane. Reference numeral 114denotes a receptive field memory for storing an image in each receptivefield when the input plane is divided into receptive fields which aresmall areas of a size equal to m×m. Reference numeral 115 denotes apolar transformation portion for applying polar transformation to theimage (receptive field image) in each receptive field, 116 denotes ahypercolumn memory for storing an image (hypercolumn image) on apolar-transformed dual plane (hypercolumn plane), and 117 denotes acharacteristic feature extraction portion for extracting characteristicfeature such as a line, a cylinder, a point, and the like.

The input plane of the image IMG of the object, having a size equal toN×N, is divided into receptive fields (IMG') which are small areas of asize equal to m×m, the polar transformation is applied to each receptivefield image, and image processing is performed based on an output(hypercolumn image) of the polar transformation to extract acharacteristic feature. The polar transformation may be polartransformation on a sphere (spherical mapping), polar transformation ona cylinder, polar transformation on a plane, or polar transformation onthe other arbitrary surface. Since the polar transformation is requiredto be applied only to the image within the receptive field of the sizeequal to m×m, and in the polar transformation on a sphere, it isrequired to draw only a great circle having a length equal to m for eachpixel in the receptive field of the size equal to m×m, so the amount ofprocessing becomes m×N². Namely, the amount of processing is remarkablyreduced compared with the conventional case wherein a great circle of alength equal to N is drawn (and the required amount of processing toN³), and the size of the hardware can also be reduced. In addition,since each receptive field can be processed independent of theprocessing for the other receptive field, interference can besuppressed. Further, since threshold values can be set independently foreach receptive field, a dark portion and a portion at which brightnessgradually varies, can be detected.

When an object is caught with an arbitrary lens by dividing an imageprojected on a predetermined projection plane, for example, a sphere(fisheye lens), a cylinder (cylindrical lens), or a plane (standard,telephoto) into receptive fields which are small areas, applying thepolar transformation to the image in each receptive field, andperforming image processing based on the polar-transformed output, highspeed processing and downsizing are possible, and interference can besuppressed.

1.2 Division into Receptive Fields

Since, in retinae and cerebra of mammals, the length of synapse whichextends from each neuron is small (less than a few millimeters), animage is processed with the image divided into small areas. Each dividedarea in a retina is called a receptive field, and each receptive fieldcontains about one thousand retinal cells. Each retinal cell in thereceptive field is connected to the visual area in the cerebrum toperform processing to extract a line segment. The visual area is dividedinto unit areas corresponding to the receptive fields, and each unitarea is called a hypercolumn. The "processing with a unit-dividedstructure" is not specific to the eye or the hypercolumn, and is ageneral structure of the cerebrum. It is considered that this shows theefficiency of the nature to reduce the total amount of synapses whichare required to perform the processing.

Taking a hint from the above, the amount of processing for extraction ofa line segment is considered below for the case where an image inputinto a camera, containing N×N pixels, is divided into units of m×mpixels. FIG. 10 is a diagram for explaining the division into receptivefields.

In FIG. 10, an input is projected onto a plane, and polar transformationis performed on a cylinder. This is the simplest polar transformation.Reference numeral 1011 denotes a plane mapping memory of N×N pixels,1012 denotes a receptive field (indicated by a small square) of m×mpixels, 1013 denotes a line segment projected on a plane, and 1014 and1015 each denote a sine wave obtained by polar transforming on acylinder two points A and B on the line segment. The two sine wavescross at a point P, and the line segment 1013 is detected as the crossedpoint by the polar transformation on the cylinder.

Since great circles (approximated as sine waves) are drawn for thenumber of the input pixels in the case where the receptive fielddivision is not performed (m=N), the amount Pm=N of the processing inthe case is given as below. ##EQU1##

On the other hand, when the image memory area of N×N pixels is dividedinto squares 1012 where the length of a side thereof is equal to m, agreat circle is drawn in each hypercolumn, and therefore, the amount ofprocessing becomes ##EQU2## Namely, the amount of processing is reducedby 1/(N/m).

Hereinafter, the above method wherein the input image is divided and thespherical mapping (precisely the polar transformation or a dualtransformation on a sphere) is performed, is denoted as a receptivefield method or a hypercolumn method. According to the conventionalmethod wherein the division of the input image is not performed, a greatcircle is drawn over the whole sphere. Since a line having an infinitelength is obtained when inversely projected into the three-dimensionalspace, the conventional method is denoted as an infinite receptive fieldmethod. To make the difference between the hypercolumn method (thereceptive field method) and the conventional method clear, thehypercolumn method (the receptive field method) may be called a "finitereceptive field method". The reason why the receptive field method isnamed as the hypercolumn method is that the structure and the functionthereof greatly resembles the hypercolumn in the primary visual cortexin the cerebrum.

1.3 Basics of Receptive Field Method

1.3.1 In Case of Plane Projection Input+Polar Transformation on Cylinder

The concrete procedure of the receptive field method according to thepresent invention is explained below for "the plane projection input+thepolar transformation (explained later) on a cylinder", where the planeprojection input+the polar transformation on a cylinder is the simplestpolar transformation. The procedure can be described such that "pixelsin each receptive field which is generated by dividing an input imageinto areas of m×m pixels (generally, the division is made so that thereis no gap between adjacent areas), are polar-transformed on a cylinder,and curves (sine waves in the case of the polar transformation on acylinder) obtained by the polar transformation are drawn on a dualplane". In FIG. 9, reference IMG denotes an image (input image) of anobject projected onto a plane, having a size of N×N pixels, IMG' denotesan image of a receptive field (receptive field image) generated bydividing the input image into small areas (receptive fields) of m×mpixels, and HIMG denotes an image on a dual plane (hypercolumn plane)generated by applying polar transformation on a cylinder to thereceptive field images (pixels A, B, . . . ).

1.3.2 In Case of Sphere Projection Input+Polar Transformation on Sphere

In the case of "the sphere projection input+the polar transformation ona sphere", the above sine waves as the curves can be changed to greatcircles. The basic steps are explained below.

(a) An image projected on a sphere is divided into small areas(receptive fields);

(b) Each pixel in the receptive field is polar-transformed on a sphere(a great circle corresponding to each pixel is drawn); and

(c) The whole receptive field is polar-transformed to a band on thesphere. Since the size of the receptive field is generally small, theband is developed on a plane. This is the dual plane (hypercolumn plane)corresponding to each receptive field.

The polar transformation in (b) can be approximated as "pixel→sinewave", and the polar transformation between the receptive field and thehypercolumn can be expressed as "each pixel in a receptive field istransformed to a corresponding sine wave on a hypercolumn". Therespective axes in the hypercolumn are provided so that a location ρ andan orientation θ of a line in a receptive field can be indicated. A linesegment in an image plane (receptive field) is detected as anintersecting point of the sine waves, and the coordinate ρ₀, θ₀ of theintersecting point indicates the location and the orientation of theline segment in the receptive field. As explained later, the above flowcan be applied in the general polar transformation, and speed-up inprocessing and suppression of interference are effected.

1.4 Cerebral Visual Cortex Model

A procedure similar to the above is operating in a primary visual cortexin a mammalian. FIG. 11A is a diagram for explaining visual informationprocessing by a mammalian and FIG. 11B for a simulation thereof. Theflow of the visual information processing in the mammalian primaryvisual cortex, from the division into receptive fields on a roundeyeball, to the transformation to the hypercolumns, can be understood asfollows.

(a) A scene is projected inside of the round eyeball to obtain a widerange of visual area almost up to 180 degrees (input of an image);

(b) Second differentiation is performed in the retina and the concentricantagonistic receptive field in the lateral geniculate nucleus (corpusgeniculatum laterale) KNE, and a contour of the image is emphasized. Thereceptive field can be expressed by connections diverging in directionsforming a cone (enhancement of a contour); and

(c) In the first visual area (hypercolumn), a line segment (a line, anedge, and a gap) in the image can be extracted for orientations with apitch of about ten degrees (extraction of a line segment).

Each neuron extends thousands of synapses, and receives signals fromthousands of synapses. Mathematically, it is considered as amulti-valued mapping. Since the multi-valued mapping is accompanied byexpansion and reduction of a dimension, the multi-valued mapping cannotbe expressed by a usual transformation such as a linear mapping.However, for extraction of a line segment, the multi-valued mapping canbe expressed by polar transformation, and a point (zero dimension) and aline (one dimension) can be transformed to each other varying theirdimensions by the polar transformation.

Thus, the modeled processor of visual information in a mammal isindicated in FIG. 11B. In FIG. 11B, reference IMI denotes an image inputportion which divides an image projected on a sphere CB into receptivefields RC, where pitches of the centers of the receptive fields and thediameters of the receptive fields are respectively equal to 16 and 16√02pixels. Reference OLD denotes a contour extraction portion which isassumed to comprise a two-dimensional DOG filter (σ=2 pixels) of the Schannel (see Wilson et al., "Quantitative Characterization of Two Typesof Linespread Function Near the Fovea", Vision Research, 18, 971-981,1978) as expressed by the equation below, and the contour extractionportion calculates a sum of products of the filter function and an inputimage to emphasize a contour.

    f(r)=exp (-r.sup.2 /σ.sup.2)-0.326 exp (r.sup.2 /(1.75σ).sup.2)

Reference PTR denotes a polar transformation portion. The receptivefield image in which the contour is emphasized is input into the polartransformation portion. The "polar transformation between the outsideknee-like body KNE and the hypercolumn cells (the receptive fieldmethod)" and "the principle of extraction of a line segment", areexplained below with reference to FIGS. 12A, 12B and 12C.

A point P in a receptive field RC is mapped into a great circle(straight line) R by the polar transformation increasing the dimension.By repeating the above process, the whole receptive field is mapped ontoa band BLT on a sphere (FIG. 12A). By developing the band on a plane(FIG. 12B), a rectangular lattice having coordinate axes correspondingto the orientation θ and the location ρ of a line, is obtained. This isthe mathematical structure of the hypercolumn, where the band BLTcorresponds to the hypercolumn, and the lattice points correspond to thehypercolumn cells.

Next, the principle of the extraction of a line segment in the receptivefield is explained below. Each cell in the receptive field ismulti-valued mapped to a plurality of lattice points (sinusoidal) in thehypercolumn. By repeating this operation, a line segment in thereceptive field is extracted as an "intersecting point of sine waves"generated by polar transformation of a point series constituting theline segment. Namely, the intersecting point Q is generated byconcentration of a line segment L (FIG. 12C) in the receptive field, andthe coordinate values θ₀, ρ₀ on the θ- and ρ-axes indicate anorientation and a location of a line segment.

Going back to FIGS. 11A and 11B, reference FDF denotes a firstdifferentiation portion. Although a "narrow line" is detected by theabove operation (the DOG filter→polar transformation), "edges(brightness variation portion)" which appear frequently in an image,cannot be detected by the above operation. In order to detect the"edge", a calculation of a sum of products with a first differentialfilter in the ρ-direction should be applied after the above operation. Amulti-filter (with a filter width/ρ-cell pitch=1, 2, 4, 8) is used fordetecting an edge at which the variation of brightness is small.

When performing a simulation wherein the size of the input image is512×512 pixels, and the size of the hypercolumn is 22 cells in theθ-direction and 22 cells in the ρ-direction, based on the above model;the following result is obtained. The response of the hypercolumn cell,obtained by applying "the DOG filter→the polar transformation→the firstdifferentiation" to the receptive field image in the circle in theoriginal image of FIG. 13 (the enlarged image thereof is indicated inFIG. 14A), is obtained as indicated in FIG. 14B. Four edges in thereceptive field are extracted as four prominent peaks P₁ to P₄.

When extracting all peaks in the whole hypercolumn, angles, locationsand lengths of line segments corresponding to the peaks are obtained.Thus, the line segments are obtained, and the image as indicated in FIG.15 is obtained. In FIG. 15, edges in the whole image are preciselyextracted. Although the original image contains portions wherein thevariations of the brightness are small, these edges are extracted due tothe application of a multi-filter as explained later.

1.5 Expansion to Various Polar Transformations

In the above, only the spherical mapping (polar transformation on asphere) is explained to simplify understanding. However, the scope ofthe present invention is not limited to spherical mapping, and generalpolar transformation can be applied to the present invention. Bydividing an arbitrary input plane into small areas, and applying variouspolar transformations thereto as explained below, the above explainedadvantages can be expected. The various variations of the polartransformation are explained below.

1.5.1 Polar Transformation in Two-Dimensional Projective Space

As long as the image processing is in consideration, an input is in atwo-dimensional surface, and polar transformation in the two-dimensionalprojective space is utilized. The polar transformation is transformationbetween "a plane containing an origin of a three-dimensional Affinespace" and "a vector (point) passing through the origin andperpendicular to the plane". The transformation in the two dimension canbe expressed by assuming an arbitrary surface and expressing anintersecting point with the surface, where a "line" in a broad sense anda point are transformed to each other, expanding/reducing the dimension.A sphere, a cylindrical surface, a plane, and the like are considered asthe above arbitrary surface.

1.5.1.1 Polar Transformation on Sphere (spherical mapping)

As indicated in FIG. 16, in "the polar transformation on a sphere(spherical mapping)", representation is made by an intersecting point ofthe vector and a unit sphere CUB having an origin O at the center, and astraight line (great circle) and a point are transformed to each other.Namely, the great circle (polar line) CIR generated by mapping onto thesphere a straight line S on a plane PL containing the origin O, ispolar-transformed to a pole P which is an intersecting point of thesphere and the normal line NL of the plane PL, and the pole P inverselypolar-transformed to the great circle CIR. In this transformation intwo-dimensional projective space, relationships are expressed by cyclicangles.

The aforementioned (FIG. 2) kinetic stereopsis is a method forthree-dimensional measurement paying attention to the polartransformation on a sphere, and the receptive field method explained asabove has a great advantage in the speed-up of processing and thesuppression of interference since a sphere as an input plane is dividedinto receptive fields, and polar transformation is applied to eachreceptive field image.

1.5.1.2 Polar Transformation on Cylinder

As indicated in FIG. 17, representation is made by an intersecting pointof the above vector and a cylinder CYR having the origin O in the centerline thereof in "the polar transformation on a cylinder", where astraight line (an intersecting line with an ellipse) and a point aretransformed to each other. Namely, the ellipse (polar line) ELPgenerated by mapping onto the cylinder a straight line S on the plane PLcontaining the origin O, is polar-transformed to a pole P which is anintersecting point of the cylinder and a normal line NL of the plane PL,and the pole P is polar-transformed to the polar line ELP.

The cylinder can be developed in parallel to the axis thereof on aplane. On the plane, a straight line (sine wave) and a point aretransformed to each other, and the polar transformation on the cylinderis different from the polar transformation on a plane which is explainednext.

1.5.1.3 Polar Transformation on Plane

As indicated in FIG. 18, representation is made by an intersecting pointwith a plane PLN which is apart from the origin O by a unit length inthe "polar transformation on a plane", and a straight line and a pointare transformed to each other. Namely, the straight line (polar line)LLN generated by mapping onto the plane PLN a straight line S on a planePL containing the origin O is polar-transformed to a pole P when anarbitrary conic SCV is set on the plane PLN, and the pole P is inverselypolar-transformed to the polar line LLN. Further, when two tangentiallines TL₁ and TL₂ of the conic SCV, respectively passing though anarbitrary point Pi on the polar line LLN, are obtained, and thecontacting points TP₁, TP₂ are connected to each other with a line, theconnecting lines for a plurality of points Pi on the polar line LLNintersect at one point. The intersecting point is the pole of thestraight line LLN.

Inversely, the polar line LLN is obtained by chaining intersectingpoints of pairs of tangential lines which make contact with the conicSCV at two points at which arbitrary straight lines passing through thepole P intersect with the conic SCV. An ellipse (conic), a circle, aparabola, and the like are examples of the conic SCV. The transformationbetween the pole and the polar line with the conic as indicated in FIG.18, is the most famous polar transformation.

1.5.1.4 Polar Transformation on Arbitrary Surface

In the above paragraphs, an explanation is given for a sphere, acylinder, and a plane as concrete polar transformation surfaces, that "aline segment can be extracted as a point, and the division intoreceptive fields has a great advantage in the speed-up in processing ofthe polar transformation". This is also true when the polartransformation surface is an ellipsoid of revolution (which is made byrotating an ellipse around the axis thereof), a hyperboloid ofrevolution (which is made by rotating a hyperbola), a paraboloid ofrevolution (which is made by rotating a parabola), and other arbitrarysurfaces.

1.5.2 Polar Transformation In N-dimensional Projective Space

In the above paragraphs, explanations are given for the polartransformation in the two-dimensional projective spaces, but thereceptive field methods according to the present invention can beapplied to general n-dimensional projective spaces. The polartransformation in an n-dimensional projective space is a transformationbetween "an n-dimensional hyperplane containing an origin in an(n+1)-dimensional Affine space" and "a vector (point) passing throughthe origin and perpendicular to the n-dimensional hyperplane".

The vector (zero dimension) passing through the origin is multi-valuedmapped to a group of all vectors (points) passing through the origin, onthe n-dimensional hyperplane containing the origin, increasing thedimension. This polar transformation corresponds to the above-mentionedtransformation between a point and a polar line. Inversely, all vectors(points) passing through the origin on the n-dimensional hyperplane aretransformed to the vector passing through the origin and perpendicularto the n-dimensional hyperplane. In the procedure of the inversetransformation, the respective points on the n-dimensional hyperplaneare multi-valued mapped to all points in planes perpendicular to vectorsconnecting the respective points with the origin. Since the intersectingline of the plane is a vector passing through the origin, then-dimensional hyperplane is mapped into a vector passing through theorigin, decreasing the dimension.

1.5.3 Synthetic Polar Transformation Combining Input Projection Surfaceand Polar Transformation Type

The above polar transformation can be applied to arbitrary types ofinputs. When treating inputs from cameras, there are three projectiontypes of image inputs in addition to the above variations of polartransformation, and various types of synthetic polar transformationexist.

1.5.3.1 Type of Input Projection Surface and Lens

An input image is made by cutting lines of sight from a center of thecamera on each of various projection planes. As the various projectionplanes, spherical projection, cylindrical surface projection, and planeprojection, and various types of lenses corresponding to the projectionplanes, are used. Table 1 indicates a relationship between the lens andthe projection surface.

Since projection onto a sphere is performed in the spherical projection,an image which is equivalent to that is obtained by the sphericalprojection can be obtained by a fisheye lens. The widest field of viewcan be obtained by the fisheye lens, which is similar to that used byanimals and fish.

The projection onto a cylindrical surface is performed in thecylindrical surface projection, an image which is equivalent to that isobtained by the cylindrical surface projection can be obtained by acylindrical lens. A wide field of view can be obtained in the angulardirections although the field of view in the direction of the axis islimited.

The projection onto a plane is performed in the plane projection, sothat an image which is equivalent to that is obtained by the cylindricalsurface projection can be obtained by a standard/telephoto lens. Since astraight line in a space corresponds to a straight line according to theplane projection different from the above other types of projection, theplane projection is widely used. However, the range of the planeprojection is narrowest.

1.5.3.2 Combination of Projection Type and Polar Transformation Type

The combinations of three polar transformation surfaces (a sphere, acylinder, and a plane) and various transformation types are possible. Aline segment in a broad sense can be extracted through the combinations.The various combinations are indicated in Tables 2 to 4.

1.5.3.2.1 Combination of Plane Projection and Polar Transformation

Table 2 is a table for explaining characteristics of various types ofpolar transformation which can be applied to the extraction of a linesegment from an (plane projected) image through a standard/telephotolens. The types of polar transformation include (a-1) polartransformation on a plane, (a-2) polar transformation on a cylinder,(a-3) polar transformation on a sphere, and (a-4) synthesized inversiontransformation on a plane.

FIG. 19 is a diagram for explaining the "plane projection+the polartransformation on a plane" of (a-1), wherein a straight line S isprojected on a projection plane PLN, the projected straight line LLN ispolar-transformed to a pole P, and the pole P is polar-transformed to astraight line LLN on the plane.

FIG. 20 is a diagram for explaining the "plane projection+the polartransformation on a cylinder" of (a-2), wherein a straight line S isprojected on a projection plane PLN, an ellipse ELP (polar line)generated by mapping a projected straight line LLN on a plane PL(containing an origin O and the projected straight line LLN) onto acylindrical surface CYR, is polar-transformed to a pole P which is anintersecting point of a normal line NL of the plane PL and thecylindrical surface CYR, and the pole P is inversely transformed to thepolar line ELP.

FIG. 21 is a diagram for explaining the "plane projection+the polartransformation on a sphere" of (a-3), wherein a straight line S isprojected on a projection plane PLN, a great circle CIR (polar line)generated by mapping a projected straight line LLN on a plane PL(containing an origin O and the projected straight line LLN) onto asphere CUB, is polar-transformed to a pole P which is an intersectingpoint of a normal line NL of the plane PL and the sphere CUB, and thepole P is inversely transformed to the polar line CIR.

In the synthesized inversion transformation of (a-4), inversiontransformation is further performed after the polar transformation,wherein polar transformation is made between "a circle passing throughan origin" and a point.

1.5.3.2.2 Combination of Cylindrical Surface Projection and PolarTransformation

Table 3 is a table for explaining characteristics of various types ofpolar transformation which can be applied to the extraction of a linesegment from an (cylindrical-surface-projected) image through astandard/telephoto lens. The types of polar transformation include (b-1)polar transformation on a plane, (b-2) polar transformation on acylinder, and (b-3) polar transformation on a sphere.

FIG. 22 is a diagram for explaining the "cylindrical surfaceprojection+the polar transformation on a sphere" of (b-3), wherein astraight line S is projected on a cylindrical surface CYR as aprojection surface, a great circle CIR (polar line) generated by mappinga projected ellipse ELP on a plane PL containing an origin O and theprojected curve (ellipse) ELP onto a sphere CUB, is polar-transformed toa pole P which is an intersecting point of a normal line NL of the planePL and the sphere CUB, and the pole P is inversely transformed to thepolar line CIR.

1.5.3.2.3 Combination of spherical mapping and Polar Transformation

Table 3 is a table for explaining characteristics of various types ofpolar transformation which can be applied to the extraction of a linesegment from an (spherically projected) image through a fisheye lens.The types of polar transformation include (c-1) polar transformation ona plane, (c-2) polar transformation on a cylinder, (c-3) polartransformation on a sphere, and (c-4) synthesized inversiontransformation on the plane.

FIG. 23 is a diagram for explaining the "sphere projection+the polartransformation on a sphere" of (c-1), wherein a straight line S isprojected on a projection plane PLN as a projection surface, theprojected straight line LLN generated by mapping a great circle CIR on aplane PL containing an origin O and the projection curve (the greatcircle) CIR onto a sphere CUB is polar-transformed to a pole P, and thepole P is polar-transformed to a straight line LLN on the plane.

FIG. 24 is a diagram for explaining the "sphere projection+the polartransformation on a cylinder" of (c-2), wherein a straight line S isprojected on a sphere CUB as a projection surface, an ellipse ELP (polarline) generated by projecting a projected great circle CIR on a plane PL(containing an origin O and the projected great circle CIR) onto acylindrical surface CYR, is polar-transformed to a pole P which is anintersecting point of a normal line NL of the plane PL and thecylindrical surface CYR, and the pole P is inversely transformed to thepolar line ELP.

In the synthesized inversion transformation of (c-4), inversiontransformation is further performed after the polar transformation,wherein polar transformation is made between "a great circle passingthrough a fixed point on the sphere" and a point.

1.5.3.2.4 Line Segment Extraction from Arbitrary Projection Image

The above explanations are given for the case where the sphere, thecylindrical surface, and the plane are used as a projection surface, andit is explained that "a line segment can be extracted as a point, andthe division into receptive fields has an advantage in the speed-up inthe processing of the polar transformation". Further, this is also truewhen the projection surface is an ellipsoid of revolution, a hyperboloidof revolution, and other arbitrary surfaces.

The above explanations are summarized as follows.

A line segment in a broad sense can be extracted from an image in abroad sense by any synthetic polar transformation;

There exists a type of polar transformation which is suitable for a lensof a camera (See Table 5);

Advantages of the speed-up and the suppression of interference can beobtained by division into receptive fields with any type of polartransformation.

Table 5 is a table indicating suitability of a lens for various types ofpolar transformation surfaces. The standard/telephoto lens is suitablefor the polar transformation on a plane, the cylindrical lens issuitable for the polar transformation on a cylinder, and the fisheyelens is suitable for the polar transformation on a sphere.

1.6 Evaluation of Receptive Field Method

The advantage of the improvement by the division into receptive fieldsare evaluated as follows.

1.6.1 Speed-up and Downsizing

1.6.1.1 Direct Advantage of Division into Receptive Fields

Assuming N=512 pixels, and m=16 pixels, the following equation isobtained from (3).

    Pm=m/Pm=N=1/(N/m)=1/32.                                    (4)

Namely, the amount of processing for extracting a line segment isgreatly reduced to 1/32 compared with the infinite receptive fieldmethod. This is because the amount of wiring is reduced by performingpolar transformation within each receptive field only. In addition,interference can be suppressed since processing of each receptive fieldis independent of the other receptive fields. The same division isperformed in the primary visual cortex in the cerebrum (the hypercolumncell) of a mammal. The amount of wiring is greatly reduced andinterference is avoided by limiting the wiring within each receptivefield.

Conventionally, convolution filters are applied to original images forimage emphasis. As explained later, it is possible to apply aone-dimensional filter after polar transformation instead of applicationof the convolution filter to an original image when the receptive fieldmethod is applied. The amount of processing when a one-dimensionalfilter is applied after polar transformation is compared below with theamount of processing when the convolution filter is applied. Since eachpoint of an image is developed on a square having sides respectivelyequal to a, the amount P_(2Dconv) of processing thereof is

    P.sub.2Dconv =(number of input elements)×(filter size)=N.sup.2 ×a.sup.2.                                           (5)

On the other hand, the amount P_(RF+1Dconv) of processing when aone-dimensional filter is applied (division into receptive fields→polartransformation →one-dimensional filter) is ##EQU3## When compared withthe equation (5),

    P.sub.RF+1Dconv /P.sub.2Dconv =(m+a)/a.sup.2

To grasp the characteristic feature of the whole image (screen), it isnecessary to extract a characteristic feature from an obscure portion,and it is desirable to be a=m. Therefore,

    P.sub.RF+1Dconv /P.sub.2Dconv =2/m.                        (7)

As described above, there is a remarkable advantage that the amount ofprocessing is smaller than the convolution filter which does not have afunction of extracting a line segment other than a function ofemphasizing an image. This is because only the one-dimensionaldevelopment (in a great circle) is required in the application of aone-dimensional filter after the polar transformation while thetwo-dimensional development (in a square) is necessary in theconvolution filter.

1.6.1.2 Speed-up by Parallel Processing

Further, parallel processing is required for speeding up of theprocessing. Since the size of a unit of the parallel processing isproportional to a length of a great circle, the size of hardwareincreases with the size of the receptive field. The maximum receptivefield is the receptive field wherein m=N, and the size of unit hardwarefor the parallel processing is proportional to N, and is very large. Inthe finite receptive field method, the size of unit hardware for theparallel processing is very small due to the smallness of m, and is1/(N/m) of that in the infinite receptive field method.

It is required to form the unit hardware for the parallel processing ina large scale integrated circuit (LSI). However, the unit hardware forthe parallel processing is too large to mount in a large scaleintegrated circuit (LSI) by the current LSI technology. On the otherhand, the unit hardware in the finite receptive field method can bereadily formed in a LSI since the size of the unit hardware is small inthe finite receptive field method as explained above. Further, since thetotal amount of processing for all the receptive fields is small, i.e.,is equal to m³, it may be possible to form the hardware for all thereceptive fields in a LSI. Summarizing the above, the receptive field issuitable for forming the hardware by LSI technology, the processing canbe made parallel without increasing hardware size, and the processingspeed can be increased by a factor m.

1.6.1.3 Synthetic Evaluation

The above factors for improvement in the speed-up and the downsizing,respectively contribute much thereto, and the synthetic effect of thefactors is even greater. The respective factors are as follows:

1. Direct Effect: The amount of processing is reduced by 1/(N/m).

2. Speed-up By Parallel Processing: The amount of processing is reducedby 1/m.

3. High Speed Filtering by One-Dimensional Filter: The amount ofprocessing is reduced by 2/m.

The synthetic amount of processing is now precisely evaluated, and iscompared with the conventional method. In the comparison, it is assumedthat "a two-dimensional filter of a size m×m is applied to an image, anda line segment is extracted by the infinite receptive field method" inthe conventional method, and that "a line segment is extracted by thefinite receptive field method, is made prominent by a one-dimensionalfilter, and is processed by LSI's with a degree of parallelism equal tom" in the synthetic improved method according to the present invention.

The respective amounts of processing of the conventional and improvedmethods are as follows. ##EQU4##

The amount of processing in the synthetic improved method=(amount ofprocessing in polar transformation+amount of processing inone-dimensional filter)/(degree of parallelism).

Since the amount of processing in the polar transformation ##EQU5## theamount of processing in the one-dimensional filter ##EQU6##

Thus, the synthetic improvement ratio is (the amount of processing inthe synthetic improved method)/(the amount of processing in theconventional method) ##EQU7##

Namely, the amount of processing is greatly reduced. In a practical caseN=512 pixels and m=16 pixels, the synthetic improvement ratio amounts to1/384. Although this is an upper limit of the reduction of the amount ofprocessing, the advantage of the finite receptive field method is verygreat. The reduction in the amount of processing reduces the size ofhardware in addition to the speed-up.

1.6.2 Suppression of Interference

The effect of the suppression of interference as the other advantage ofthe finite receptive field method is evaluated next.

The cause of the interference is that "when another great circle islocated in a direction near a direction of a great circle correspondingto a line segment to be extracted by the spherical mapping, these greatcircles interfere with each other to deteriorate the accuracy of theextraction of the line segment".

Regarding the above object, no interference occurs according to thereceptive field method because an input image is divided into receptivefields, and a great circle other than the great circle of interest isnot contained in a receptive field of interest.

The advantage in the above "division into receptive fields+sphericalmapping" also exists in the other arbitrary polar transformation otherthan the spherical mapping (polar transformation on a sphere).

1.6.3 Evaluation of Indirect Effect of Receptive Field Method

The speed-up due to the receptive field method contributes toimprovement in the image processing function.

1.6.3.1. Processing of Obscure Image

Generally, input images may contain a line and an edge at whichbrightness gradually varies, and these line and edge may be important.However, it is difficult to detect a portion at which brightnessgradually varies, by the conventional convolution filter since theconventional convolution filter is a two-dimensional filter, the amountof processing rapidly increases with the square of the size of thefilter size, and the increase makes the processing by the filter of alarge size difficult.

According to the receptive field method, "the effect of atwo-dimensional filter can be obtained by the application of aone-dimensional filter". Therefore, a filter of a large size can beapplied with a small amount of processing. The size of 5×5 pixels is apractical upper limit of the two-dimensional convolution filter, whilefiltering of a diameter up to 13 pixels can be applied with the sameamount of processing.

Namely, a filter much greater than the size of the conventional filtercan be applied to images, and therefore a portion at which brightnessgradually varies, can be extracted precisely.

1.6.3.2 Processing of Image with Low Contrast

Generally, input images may contain a dark portion due to a lightingcondition and a portion at which brightness gradually varies. Accordingto the conventional image processing, a uniform threshold value isapplied to the whole image. Therefore, the dark portion and the portionat which brightness gradually varies, are eliminated through the imageprocessing.

According to the receptive field method, areas of m×m pixels areprocessed independently of each other, and a uniform threshold is notapplied to the whole image. Therefore, the above drawback does notexist, and the dark portion and the portion at which brightnessgradually varies, can be detected. Since each receptive field is small,the lighting condition can be considered to be uniform within eachreceptive field. Therefore, a uniform threshold value can be appliedwithin each receptive field.

Namely, according to the receptive field method, when an input imagecontains a dark portion due to a lighting condition and a portion atwhich brightness gradually varies, these portions can be detectedprecisely, while the conventional threshold processing cannot detectthese portions.

1.6.3.3 Insensitive Lighting Condition

In the conventional image processing, a characteristic feature extractedfrom an image is seriously affected when the intensity and the directionof lighting vary. This causes a serious problem when the function isequipped in a robot working outside, where the brightness is affected byweather. The problem is caused because representation of a thresholdvalue independent of the variation of lighting is difficult in theconventional image processing.

Since an image is processed by being divided into small areas accordingto the present invention, a parameter (for example, a half value of apeak value of an output processed within an area) independent oflighting can be used as a threshold value, and therefore extraction of acharacteristic feature which is not affected by the variation oflighting can be achieved. Thus, according to the receptive field method,characteristic features such as a line segment can be stably extractedwhen the intensity and the direction of the lighting vary.

1.7 Simulation

FIGS. 25A, 25B, and 26 show a result of the simulation wherein aspherical mapping image (an image shot through a fisheye lens) of N×Npixels (N=512 pixels), spherically projected as in FIG. 16, is dividedinto receptive fields of m×m pixels (m=16) according to the receptivefield method, polar transformation is applied on a sphere to each pointin each receptive field by the polar transformation circuit 113, and aline segment in the receptive field is extracted by extracting a peakvalue from an output hypercolumn. The original image is the same asindicated in FIG. 13, FIG. 25A is a magnification of a portion which isencircled by a circle in FIG. 13, FIG. 25B shows responses in theelement receptive field (hypercolumn) corresponding to the portionencircled by contour lines, and FIG. 26 shows responses of all of thereceptive fields (regenerated image).

As indicated in FIG. 25B, two lines C1 and C2 (see FIG. 25A),respectively in the horizontal and vertical directions, arepolar-transformed, and are respectively extracted as two sharp peaks P1and P2 corresponding to the two lines in the hypercolumn memory.

Further, as indicated in FIG. 26, the whole image can be stablyregenerated from peaks extracted in the respective receptive fields, andlow contrast portion and obscure portions can be stably extracted.

Although, in the above explanations, an image input mainly through acamera or a lens, is divided into small areas, polar transformation isperformed in each small area, and image processing is performed based onthe result of the polar transformation, the present invention can beapplied to arbitrary images. For example, the hypercolumn image may befurther divided into small areas, polar transformation may be performedin each small area, and image processing may be performed based on theresult of the polar transformation.

Although, in the above explanations, the density of pixels is assumed tobe uniform, the sizes of the receptive fields may be varied according tothe densities of pixels in the respective receptive fields when thedensity of pixels varies with the locations (at the center, near theperiphery, and the like) in the image so that the numbers of therespective receptive fields can be the same. The hypercolumns of mammalsare constructed in a manner similar to above.

1.8 Requirement for Filtering

Although a line segment can be extracted quickly from a wide field ofview image by the receptive field method according to the presentinvention, there are the following requirements regarding the filteringwhen considering quality of the input image and the characteristicfeature to be extracted.

1.8.1 Precise Extraction of Line Segment from Obscure Image

When moving in a narrow environment such as an industrial plant, awarehouse, or the like, it is necessary to recognize the wholeenvironment with a wide field of view. In this case, it is difficult toobtain the whole image in focus due to the wide field of view, andtherefore the image may contain an obscure portion. Further, objects areoften chamfered for safety, and the chamfered ridge-line will appear asan obscure edge or line in the image.

However, information on the obscure portion is very important for movingor working surely, and therefore, the function of extracting features ofboth the sharp portions and obscure portions in the input image isrequired. Although filtering with a large mask size is necessary forextracting a line segment from an obscure image, it is impossible by theapplication of the conventional two-dimensional filter.

1.8.2 Recognition of Type of Line Segment

The term "line segment" used in the above explanations can have threedifferent meanings: lines, edges, and gaps. In the mammalian cerebrum,cells for recognizing a line, an edge, and a gap, are provided,respectively.

1) Line: This is a luminous band.

2) Edge: This is a border between a luminous portion and a dark portion,and the edge is the major feature in actual images.

3) Gap: This is an inverse of the line, i.e., a dark band.

For example, the above three types of line segments may correspond to"piping of a small diameter", "steps", "a shadow portion of the pipingof a small diameter", and the like in an industrial plant, and these areimportant features for moving and working therein. According to thespherical mapping method described above, the "line" and the "gap" canbe extracted. Although filtering for emphasizing edges is necessary toextract the "edge", which is the major feature in the actual image, suchfiltering has not been realized at present. Further, it is desirable toapply filtering to the "line" and the "gap" for making these portionsprominent. However, such filtering has not been realized because theapplication of a filter of a large size is difficult due to thenecessity of the large amount of calculation and processing time.

The mammalian cerebrum has cells for extracting the above three types ofline segments (lines, edges, and gaps) for precisely sorting andrecognizing information necessary for movement.

1.8.3 Conventional Technique of Filtering

The above problems relate to the filtering, and are summarized asfollows.

Filtering of a large size mask which is necessary for processing ofobscure image; and

Filtering for separately extracting the three types of line segments.

The conventional techniques for the above are as follows.

1.8.3.1 Conventional Filtering of a Large Size Mask

In the conventional image processing, the emphasizing of contours andextraction of the features are performed by applying a two-dimensionalconvolution filter to an image within a mask. The amount of processingis estimated to be m² ×N², which increases m², when the size of the maskis m×m, and the size of the image is N×N, since convolution calculationof m² is required for each pixel. Therefore, the upper limit of the masksize in the conventional technique is about 5×5 pixels from the point ofview of the processing speed, while a mask of 10×10 pixels will berequired for processing of the obscure image. Namely, the processing ofthe obscure image is difficult by the conventional technique.

1.8.3.2 Filtering Technique for Line Segment

Conventionally, the line segments are extracted by the Houghtransformation or the projection method after applying thetwo-dimensional Gaussian filter. This method has the same problem as theabove technique since the two-dimensional Gaussian filter is used, andthe amount of processing rapidly increases with the square of the filtersize.

Although, as an alternative, there is a method of applying small filtersin series to synthesize a large filter, the total necessary amount ofprocessing in this case is m² ×N² except for a case where the filterfunction is special, and the problem is essentially not solved.

As explained above, there is a problem that the amount of processingincreases due to the filtering process in the process for extraction ofa line segment.

In addition, the use of a large mask is impossible in the conventionaltechnique due to the increase in the amount of processing, therefore theprocessing of an obscure image, which requires a large mask, isdifficult, and the precise extraction of a line segment from the obscureimage is impossible.

Further, the edge cannot be extracted by the conventional technique, andthe lines and the gaps cannot be extracted as prominent features.

2. Second Aspect of Present Invention

2.1 Basic Construction of Second Aspect of Present Invention

FIG. 27 is a diagram illustrating the basic construction of the secondaspect (the receptive field method +one-dimensional filtering) of thepresent invention.

In FIG. 27: reference numeral 212 denotes an input memory for storing animage of an object (input image) IMG having a size equal to N×N pixelsand being projected on a predetermined input plane; 214 denotes areceptive field memory for storing an image (receptive field image) ineach receptive field when the input plane is divided into receptivefields (small areas) of a size m×m; 215 denotes a polar transformationportion for applying polar transformation to each receptive field; 217denotes a one-dimensional filter circuit for applying one-dimensionalfilter processing to an output of the polar transformation; 219 denotesa hypercolumn memory for storing an image on a dual plane (hypercolumnplane), to which the polar transformation and the one-dimensionalfiltering is applied; and 221 denotes a characteristic featureextraction portion for extracting a characteristic feature such as aline, an edge, and a gap in an image.

The image IMG of the object of the size equal to N×N pixels is dividedinto receptive field images of the size equal to m×m pixels, and thedivided images are stored in turn in the receptive field memory 214. Thepolar transformation portion 215 performs polar transformation on eachpolar transformation image. The one-dimensional filter circuit 217applies a predetermined one-dimensional filter processing to the outputof the polar transformation, and stores the output of the processing(hypercolumn images) in the hypercolumn memory 219. The characteristicfeature extraction portion 221 performs image processing based on thehypercolumn image to extract a characteristic feature such as a line, anedge, and a gap. Since only a one-dimensional filter is required to beapplied in the present invention, the amount of processing is remarkablyreduced compared with the conventional case wherein the two-dimensionalfilter is applied, a large filter which enables processing of obscureimages and precise filtering, can be applied with the same amount ofprocessing as the conventional case, and therefore a line segment can beprecisely extracted from an obscure image.

In addition, an edge can be extracted while the extraction of an edge isdifficult in the conventional two-dimensional filter method. Since thetwo-dimensional filter for extracting an edge is an odd-function filter,and different filters must be provided for the respective directions,the amount of processing is greatly increased compared with the amountof processing for extracting a line. According to the present invention,as explained later, arbitrary filtering can be realized by "aone-dimensional filter (ρ-direction) independent of θ" after the polartransformation, and the extraction of an edge, for which an odd-functionis required, can be performed with an amount of processing which is notso large. The one-dimensional filter is applied after the projection inall of the directions (polar transformation). Namely, the isotropicprocessing is already performed. The one-dimensional filtering appliedafter the polar transformation does not affect the isotropy, andtherefore an arbitrary filter can be applied. Thus, a one-dimensionalodd-function filter which is necessary for the extraction of an edge isapplied as the one-dimensional filter, and an edge can be extractedwhile, conventionally, the edge cannot be detected.

Further, when the one-dimensional filter is constituted by a skeletonfilter the characteristic of which can be represented by the Dirac'sδ-function, an edge can be extracted by a simple and fast calculationprocess.

As another variation, when a plurality of types of one-dimensionalfilter processing having different widths are applied to the output ofthe polar transformation at the same time, and the outputs thereof aresynthesized (multi-filter), obscure portions and sharp portions whichexist in the same image can be extracted at the same time to enablemovement and working based on recognition of the whole image.

In addition, the image of an object is divided into receptive fields,polar transformation is performed on each receptive field image, andone-dimensional filter processing is applied to the polar-transformedoutput so that a line can be extracted. Further, a gap can be extractedby inversing the polarity of the output of the one-dimensionalfiltering, and an output of a one-dimensional Gaussian filter in avicinity of a peak in the output of the polar transformation isselected, so that exclusive extraction of a line segment(s) becomespossible.

Further, when the image of an object is divided into receptive fields,polar transformation is performed on each receptive field image, and aone-dimensional gradient filter processing is applied to the output ofthe polar transformation, so that an edge can be extracted.Alternatively, when a one-dimensional Gaussian filter processing isapplied in addition to the above one-dimensional gradient filterprocessing, the edge can also be extracted. Further, when a plurality oftypes of one-dimensional gradient filter processing having differentwidths are applied to the output of the polar transformation at the sametime, and the outputs thereof are synthesized (multi-filter), obscureportions and sharp portions which exist in the same image can beextracted at the same time. Alternatively, a plurality of types ofone-dimensional Gaussian filter processing may be applied in addition tothe above plurality of types of one-dimensional gradient filterprocessing for extracting the edge.

As another variation, in a one-dimensional multi-stage filter which isconstituted by connecting a one-dimensional gradient filter and aone-dimensional Gaussian filter in multiple stages, an output of eachstage is separated into positive and negative signals to be subjected tothe filter processing in the following stage(s). Thus, fine featuressuch as a narrow band, dense lines, an edge, and the like, can beextracted while suppressing interference.

2.2 Outline of One-Dimensional Filter

It is desirable to precisely separate and extract the three types ofline segments from obscure portions. However, since the convolutionfilter for extracting a line segment in the conventional techniqueperforms two-dimensional calculations, the amount of processingincreases with the square of the filter size, and therefore theprovision of a large filter for extraction of an obscure portion isdifficult. In the present invention, attention is directed to thereceptive field method to perform the function of the two-dimensionalfilter by a one-dimensional filter (one-dimensional filter method). Theessential points of the one-dimensional filter method are explainedbelow. Namely, the principle of the one-dimensional filter method isbased on the relationship that "to perform polar transformation after atwo-dimensional convolution filter is applied" is equivalent to "toapply a one-dimensional filter after polar transformation is performed".

Due to the above relationship, the function of the two-dimensionalfilter can be performed by a one-dimensional filter, and the amount ofprocessing is reduced to about 2/(the diameter of the convolutionfilter) as explained later. Thus, the function of the two-dimensionalfilter of 13×13 pixels can be performed with an amount of processingwhich is equal to the amount of processing of the two-dimensional filterof 5×5 pixels, and a filter which is large enough to perform theprocessing of obscure images and the fine filtering, can be realized.

The above important relationship that "the function of thetwo-dimensional filter can be performed by a one-dimensional filter" isderived from the "polar transformation" in the receptive field method.This relationship can be widely utilized in the image processing, suchas for quickly emphasizing and extracting arbitrary variations ofbrightness, and the like. The basic flow of processing is: inputimage→the receptive field method (division into receptive fields→polartransformation)→one-dimensional filter.

2.3 Principle and Characteristic of One-Dimensional Filter Method

Although the receptive field method can be applied in general polartransformation, for ease in understanding the principle, the principleof the one-dimensional filter method is explained below in relation tothe projection method which is a kind of polar transformation.

2.3.1 Principle of One-Dimensional Filter Method

FIGS. 28A, 28B, 28C, 28D, 28E, 28F, and 28G are diagrams for explaininga principle of the one-dimensional filter method in the presentinvention. First, symbols used in the explanation are explained. In theexplanation, ψ denotes a projection axis which makes an angle θ to they-axis of the x-y coordinate system, and ρ denotes a coordinate axisperpendicular to the projection axis ψ. When an original image isdenoted by f(ρ,ψ), and the two-dimensional convolution filter is denotedby g(ρ,ψ), the output c(ρ,ψ) of the two-dimensional convolution filteris expressed as

    c(ρ,ψ)=∫∫g(ρ-α,ψ-β)f(α,β)d.alpha.dβ.                                             (11)

An output C(ρ,ψ) of the projection of c(ρ,ψ) in the ψ-direction isexpressed as ##EQU8## When defining ##EQU9## the output C(ρ,ψ) of theprojection of the two-dimensional convolution in the ψ-direction isfinally expressed as

    C(ρ,ψ)=∫G(ρ-α)F(α)dα,   (14)

where F(ρ,ψ) and G(ρ,ψ) are projection results of the original image andthe two-dimensional convolution in the ψ-direction, respectively.Therefore, from the equation (14),

    C(ρ,ψ)=one-dimensional convolution of "the projection component of the original image" and "the projection component of the filter",(15)

is derived. From the equation (14), the above-mentioned relationshipthat "to perform polar transformation after a two-dimensionalconvolution filter is applied" is equivalent to

    "to apply a one-dimensional filter after polar transformation is performed",(16)

is proved.

According to the above relationship, when the projection processing isperformed preceding the filtering processing, the filtering process canbe the application of a one-dimensional filter in the ρ-direction, andthe increase in the amount of processing as in the two-dimensionalconvolution can be avoided. The image of the above mathematicaltransformation can be understood by FIGS. 28A-28G, where FIG. 28Aindicates an original image, FIG. 28B indicates the filtercharacteristic of the two-dimensional convolution filter, FIG. 28Cindicates the result of the application of the two-dimensionalconvolution filter to the original image, FIG. 28D indicates the outputC(ρ,ψ) of the projection in the ψ-direction of the two-dimensionalconvolution, FIG. 28E indicates the projection component of the originalimage in the ψ-direction, FIG. 28F indicates the one-dimensionalconvolution filter (the projection component of the two-dimensionalfilter characteristic), and FIG. 28G indicates the result of theone-dimensional convolution of "the projection component of the originalimage" and "the projection component of the filter". The results ofFIGS. 28D and 28G are equal, and this proves the relationship that "toperform polar transformation after a two-dimensional convolution filteris applied" is equivalent to "to apply a one-dimensional filter afterpolar transformation is performed". In FIGS. 28A to 28G, ⊙ denotes aconvolution calculation, and P0() denotes a projection to the directionwhich makes an angle θ to the y-axis.

Although the above proof is based on the projection method for ease inunderstanding, the relationship exists for general polar transformation,where the "polar transformation" is a generalized concept of the"projection". Therefore, for general polar transformation, "to performpolar transformation after a two-dimensional convolution filter isapplied" is equivalent to "to apply a one dimensional filter after polartransformation is performed". Thus, the filtering processing can be theapplication of a one-dimensional filter for general polartransformation, and the high speed processing with a filter of a largesize is possible.

2.3.2 Limit of Two-Dimensional Convolution Filter

It is important to note that the object of the two-dimensionalconvolution filter is to filter the brightness of the input image,transform the brightness to a scalar quantity, and perform the emphasisof contours and the like. However, generally, the output of the equation(11) is transformed to a function of the angle θ as a parameter, inaddition to ρ and ψ. Exactly, the equation (11) is expressed as

    c(ρ,ψ,θ)=∫∫g(ρ-α,ψ-β,θ)f(.alpha.,β)dαdβ                              (17)

The filter function g(ρ,ψ,θ) is a function of the location (ρ,ψ) of thepixel and the angle θ of the projection axis, and has three degrees offreedom. The equation (17) is a scalar only when the filter functiong(ρ,ψ,θ) is not a function of θ, i.e., the filter function g(ρ,ψ,θ) isisotropic. In this case, the filter function is a "ring-like function"having a common center of the receptive field. As explained later indetail, there is a limit in that this ring-like function can performtwo-dimensional differentiation of even order only.

Thus, it is understood that the two-dimensional convolution filterswhich can perform transformation of brightness, required in imageprocessing, are limited to a combination of ring-like two-dimensionalfilters, and only the two-dimensional differentiation of the even ordercan be performed.

In the differentiation of the even order, there is a limit that the"edge (a border at which the variation of brightness is locallymaximum)", which is an important feature, cannot be extracted. On theother hand, as explained later, an arbitrary filter can be applied inthe one-dimensional filter method, and extraction of an "edge" ispossible.

2.4 Actual Proof

The "equivalence of the two-dimensional filter to the one-dimensionalfilter" is proven by simulation as follows.

2.4.1 Two-Dimensional Convolution+Receptive Field Method

As "the two-dimensional convolution+the receptive field method", theflow of:

    input image→the two-dimensional convolution→the receptive field method (division into receptive fields→polar transformation),(18)

is adopted, and

the size of the receptive field=a diameter of 22 pixels, and

the two-dimensional filter=two-dimensional convolution filter processedin the retina of the mammalian (σ=1 pixel)

    g(r)=exp (-r.sup.2 /σ.sup.2)-0.236 exp (-r.sup.2 /(1.75σ).sup.2),                                    (19)

are used as parameters.

A response on the dual plane (hypercolumn plane), which is obtained byapplying the processing of the flow of (18), to the receptive fieldimage (see FIGS. 29A and 29B) in the round receptive field CRC cut outfrom the original image, is indicated by contour lines in FIGS. 29C and29D, where a line L in the receptive field is extracted as a prominentpeak PK in the dual plane. The curves in FIGS. 29B and 29D indicatebrightness distribution in the horizontal cross-section in FIGS. 29A and29C, respectively.

2.4.2 Receptive Field Method+One-Dimensional Convolution

As "the receptive field method+the one-dimensional convolution", theflow of:

    input image→the receptive field method (division into receptive fields→polar transformation)→the one-dimensional convolution,(20)

is adopted, and

the size of the receptive field=a diameter of 22 pixels, and

the one-dimensional filter=one-dimensional filter (σ=1 pixel) generatedby projecting the two-dimensional filter of (19) in the ψ-direction

    g(ρ)=exp (-ρ.sup.2 /σ.sup.2)-0.57 exp (-ρ.sup.2 /(1.75σ).sup.2),                                    (21)

are used as parameters.

A response on the dual plane (hypercolumn plane), which is obtained byapplying the processing of the flow of (20), to the receptive fieldimage (see FIGS. 30A and 30B) in the round receptive field CRC cut outfrom the original image, is indicated by contour lines in FIGS. 30C and30D, where a line L in the receptive field is extracted as a prominentpeak PK in the dual plane.

When comparing the simulation result of the "the two-dimensionalconvolution+the receptive field method" (FIGS. 29A to 29D) with thesimulation result of "the receptive field method+the one-dimensionalconvolution" (FIGS. 30A to 30D), it is understood that there is nosignificant difference between these results. Namely, the aboverelationship explained before as the principle of the one-dimensionalfilter method is proven.

2.5 Amount of Processing and Processing System

Next, the amount of processing is estimated in comparison with thetwo-dimensional convolution filter.

2.5.1 Amount of Processing of "Division into Areas+PolarTransformation+One-Dimensional Filter"

When a denotes a width of a one-dimensional filter, m denotes a size ofthe receptive field, and N denotes a size of an image, the amount ofP_(1D) of processing is

    P.sub.1D ={amount of processing of polar transformation}+{amount of processing of one-dimensional filter}.

Since the amount of processing ##EQU10## the amount of processing of theone-dimensional filter ##EQU11## Further, since a has a size near m,"the amount of processing of the polar transformation is approximatelythe same as the amount of processing of the one-dimensional filter". Inaddition, since the filter size does not exceed m,

    P.sub.1D ≦2mN.sup.2.                                (25)

2.5.2 Amount of Processing of "Two-Dimensional Filter+Division intoAreas+Polar Transformation"

When the diameter is denoted by a, the amount P_(2D) of processing of"two-dimensional filter+division into areas+polar transformation" isexpressed as ##EQU12##

Since the amount of processing of polar transformation=the equation(22)=mN², ##EQU13## the filter size a does not exceed m,

    P.sub.2D ≦(m.sup.2 +m)N.sup.2                       (28)

    ≈m.sup.2 N.sup.2.                                  (29)

2.5.3 Comparison of Amounts of Processing

2.5.3.1 Total Comparison of Amounts of Processing

From the equations (14) and (17),

    P.sub.2D /P.sub.1D =(a.sup.2 +m)/(m+a).                    (30)

The amounts are equal when a=1, and the ratio increases with a. At themaximum filter width a=m, ##EQU14## Therefore, a great contribution isobtained in that the one-dimensional filter can perform the processingof a large filter equivalent to the two-dimensional filter of (m²/2)×(m² /2) with the amount of processing of the two-dimensional filterm×m when the filter width is equal to the maximum (a=m).

2.5.3.2 Comparison of Amounts of Processing

From the equations (23) and (26),

    F.sub.2D /F.sub.1D =a                                      (34)

is obtained, and the amount of processing increases in proportion to thefilter width a. The reason is because the amount of processing in thetwo-dimensional filter increases with the square of the diameter of thefilter while the amount of processing in the one-dimensional filterincreases in proportion to the filter length, and

    "the processing of a large one-dimensional filter of a size a.sup.2 ×a.sup.2 can be performed with the amount of processing of the two-dimensional filter of a size a×a".              (35)

2.5.4 Processing System

The above processing can be performed in accordance with the followingfour types of flows. FIGS. 31A, 31B, 31C, and 31D are block diagramsillustrating constructions performing these processing flows, where FIG.31A is a block diagram for the case of "performing the polartransformation and the one-dimensional filter in the same processingapparatus"; FIG. 31C is a block diagram for the case of "performing thepolar transformation and the one-dimensional filter in differentprocessing apparatuses"; FIG. 31B is a block diagram for the case of"performing the two-dimensional filter and the polar transformation inthe same processing apparatus"; and FIG. 31D is a block diagram for thecase of "performing the two-dimensional filter and the polartransformation in different processing apparatuses".

In Table 6, these four-types of processing systems are compared withregard to the processing time and the hardware size, where thecomparison is made for the maximum filter size (a=m). Although thehardware size for the case where the polar transformation and theone-dimensional filter are performed in the same processing apparatus ishalf of the hardware size for the case where the polar transformationand the one-dimensional filter are performed in different processingapparatuses, the processing time in the case where the polartransformation and the one-dimensional filter are performed in the sameprocessing apparatus is two times the processing time in the case wherethe polar transformation and the one-dimensional filter are performed indifferent processing apparatuses since the filter processing and thepolar transformation cannot be performed at the same time in the lattercase. It is understood that the one-dimensional filter method issuperior to the two-dimensional filter method in both the cases wherethe above processing is performed in the same apparatus and in differentapparatuses.

2.6 Construction of Line Segment Extraction System

2.6.1 Overall Construction

FIG. 32 is a diagram illustrating a construction of a line segmentextraction system as an embodiment of the second aspect of the presentinvention, where the line segment extraction is performed as: an inputimage→the receptive field method (division into receptive fields→polartransformation)→the one-dimensional convolution. The construction ofFIG. 32 is also a construction of a line segment extraction system as anembodiment of the first aspect of the present invention.

In FIG. 32: reference numeral 10 denotes a camera for inputting animage; 11 denotes a control portion for controlling the overall systemfor line segment extraction; 12 denotes an input memory for storing animage of an object (input image) having a size equal to N×N pixels andbeing projected on a predetermined input plane; 13 denotes a receptivefield cutting-out circuit for cutting out in turn images (receptivefield image) in the respective receptive fields which are obtained bydividing the input plane into small areas of a size m×m; 14 denotes areceptive field memory for storing an image (receptive field image) ineach receptive field; 15 denotes a polar transformation circuit forapplying polar transformation in each receptive field; and 16 denotes ahypercolumn memory for storing an image (hypercolumn image) on a dualplane (hypercolumn plane) obtained by the polar transformation. Thehypercolumn memory 16 is constituted by storage-area hypercolumn cellscontaining ρ_(max) cells in the ρ-direction and θ_(max) cells in theθ-direction and ρ_(max) ×θ_(max) cells as a total.

Reference numeral 17 denotes a one-dimensional filter circuit forapplying one-dimensional filter processing to the hypercolumn imageobtained by the polar transformation; 18 denotes an element hypercolumnmemory for storing a hypercolumn image to which the filter processing isapplied; 19 denotes a total hypercolumn memory for storing a totalhypercolumn image; 20 denotes a transfer circuit for transferring thehypercolumn image in the element hypercolumn memory 18 to apredetermined storage area in the total hypercolumn memory 19; and 21denotes a characteristic feature extraction portion for extracting afeature of a line segment such as a line, an edge, and a gap in animage.

The polar transformation circuit 15 performs the polar transformation oneach pixel in the receptive field, i.e., transforms each pixel to agreat circle or the like, corresponding to the pixel, and stores thegreat circle in the polar transformation hypercolumn memory 16. Inpractice, the polar transformation is approximated as "transformationfrom a pixel to a sine wave" due to small size of the receptive field,and each pixel in the receptive field is transformed to a sine wave,corresponding to the pixel, on the hypercolumn, and stores the sine wavein the polar transformation hypercolumn memory 16.

The one-dimensional filter circuit 17 is provided for performing theextraction of a line, edge, and gap with a small amount of processing.Conventionally, for emphasizing a contour, image data is filtered by thetwo-dimensional convolution filter, and is then polar-transformed.However, according to the convolution method, the amount a² ofprocessing is necessary when the filter size is equal to a, and theamount of processing increases with the increase in the filter size.However, since "two-dimensional convolution filter+polar transformation"is equivalent to "application of a one-dimensional filter in theρ-direction after polar transformation", the one-dimensional filter isapplied after the polar transformation in the embodiment of FIG. 32.Thus, the amount of processing is reduced about 2/a of the amount ofprocessing in the convolution method.

2.6.2 Polar Transformation Circuit

As indicated in FIG. 33, the polar transformation circuit comprises: areading control portion 15a for reading out the brightness of thereceptive field image (amplitude) pixel by pixel from a correspondingaddress in the receptive field memory 14; a polar transformation portion15b for performing the polar transformation for each pixel(transformation from the address of the pixel to an address for the sinewave); and a writing control portion 15c for writing the amplitude readas above at a plurality of storage positions in the polar transformationhypercolumn memory 16, where the storage position is indicated by theaddress of the sine wave obtained by the polar transformation.

The polar transformation portion 15b comprises: an addresstransformation memory 15b-1 which provides correspondence between apixel address of the receptive field memory 14 and a plurality ofaddresses in the polar transformation hypercolumn memory 16; and anaddress transformation circuit 15b-2 which transforms the address in thereceptive field memory 14 to an address in the polar transformationhypercolumn memory 16. The address transformation table stored in theaddress transformation table memory 15b-1 is provided for applying polartransformation to each point in the receptive field to transform thepoint to a sine wave on the hypercolumn plane. That is, the addresstransformation table memory 15b-1 transforms an address corresponding tothe point in the receptive field to a plurality of addresses on thepolar transformation hypercolumn memory, where a series of pointsconstituting the sine wave is located in the plurality of addresses.

The reading control portion 15a reads amplitude data from the firstaddress in the receptive field memory 14, and inputs the amplitude dataand the address data (the first address) into the polar transformationportion 15b. The polar transformation portion 15b transforms the firstaddress in the receptive field memory to a plurality of addresses in thepolar transformation hypercolumn memory 16 for the series of points ofthe sine wave, and outputs the addresses and the amplitude. The writingcontrol portion 15c adds the amplitude data input thereto to the content(the initial value thereof is equal to zero) of each address in thepolar transformation hypercolumn memory 16, where the address is inputthereto at the same time as the amplitude data, and writes the addeddata in the address. Thereafter, the above processing is performed forall the addresses in the receptive field memory 14 to complete the polartransformation for the receptive field image.

2.6.3 One-Dimension Filter Circuit

As indicated in FIG. 34, the one-dimensional filter circuit 17comprises: a reading control portion 17a for reading out amplitudes fromaddresses (for example, as indicated in the hatched portion in FIG. 34)for a constant θ in the polar transformation hypercolumn memory 16,where the number of the addresses equals ρ_(max), and outputting theamplitudes; a one-dimensional filter portion 17b for applyingone-dimensional filter processing to the respective amplitudes read outas above; and a writing control portion 17c for writing the result ofthe one-dimensional filter processing into a storage area of the elementhypercolumn memory 18.

The one-dimensional filter portion 17b is constituted by aone-dimensional filter circuit 17b-1 and a sum-of-product circuit 17b-2.In the one-dimensional filter circuit 17b-1, the characteristic of theone-dimensional filter is stored in the form of discrete values, whereinthe ρ-axis is used as a horizontal axis. Namely, the filtercharacteristic values are stored for the respective locations -ρ_(max)/2˜ρ_(max) /2 on the ρ-axis in the one-dimensional filter circuit 17b-1.For example, in the case where the one-dimensional filter is aone-dimensional first differential filter, the characteristic values onthe curve as indicated in FIG. 35A are stored, and in the case where theone-dimensional filter is a one-dimensional second differential filter,the characteristic values on the curve as indicated in FIG. 35B arestored. The width and values of the filter are appropriately determinedas needed. The sum-of-product circuit 17b-2 multiplies the ρ_(max)values of amplitudes read out as above by the corresponding ρ_(max)characteristic values stored in the one-dimensional first memory 17b-1,respectively to output a sum of the products (as an amplitude), andperforms a similar calculation of sums of products for the cases whereinthe correspondence between the pixels in the values of amplitudes andthe pixels of the characteristic values are shifted, to output theresults thereof.

For example, ρ_(max) values of amplitudes are read out from alladdresses (ρ_(max) values of addresses) in the polar transformationhypercolumn memory 16 for the orientation θ_(i) (the initial value is_(i) =1). Next, the location of the filter characteristic value for ρ=0is made to correspond to the address A_(0i) located at the left endamong the ρ_(max) values of addresses. Then, the multiplicationoperations of the corresponding values of the amplitudes and thecharacteristic values are performed, and a sum of the products isobtained by calculation to write the sum in the address A_(0i) in theelement hypercolumn memory 18 by the writing control portion 17c.

When the above operation is completed, the location of the filtercharacteristic value for ρ=0 is made to correspond to the address A_(1i)located at the second from the left end among the ρ_(max) values ofaddresses. Then, the multiplication operations of the correspondingvalues of the amplitudes and the characteristic values are performed,and a sum of the products is obtained by calculation to write the sum inthe address A_(1i) in the element hypercolumn memory 18 by the writingcontrol portion 17c. A similar operation is performed thereafter untilthe location of the filter characteristic value for ρ=0 is madecorresponding to the address A_(max),i located at the right end amongthe ρ_(max) values of addresses. Then, the index _(i) is shifted to _(i)+1 to repeat the above calculation of the sum of products for theone-dimensional filtering.

The above calculation of the sum of products is expressed by equationsas follows.

Among the area Aρi in the hypercolumn memory, data in the rowcorresponding to the orientation θ_(i) are denoted by A(0-ρ_(max)/2,_(i)), A(1-ρ_(max) /2,_(i)), . . . A(ρ,_(i)), . . . A(ρ_(max)/2,_(i)), and the content of the one-dimensional filter memory isdenoted by g(0-ρ_(max),i), g(1-ρ_(max),i), . . . g(ρ,_(i)), . . .g(ρ_(max-1),i), g(ρ_(max),i), the output in each orientation θ_(i) fromthe one-dimensional filter portion is given as

    OUT(ρ,θ.sub.i)=Σ.sub.k A(k,i)g(ρ-k,i)

where the range of the accumulation is 0˜ρ_(max) +1.

2.7 Evaluation of "Receptive Field Method+One-Dimensional Filter"

2.7.1 Speed-up of Large Mask Filter

By the "receptive field method+one-dimensional filter", the function ofthe two-dimensional filter can be performed by a one-dimensional filter,and processing of a large mask can be performed at high speed. When thefilter size of the conventional method is assumed to be m×m, the amountof processing is reduced to 2/m, and the greater the effect of thereduction of the amount of processing is, the larger the filter size is.For example, processing of a two-dimensional filter having a size of13×13 pixels can be effectively performed with the same amount ofprocessing of the conventional two-dimensional filter of a size of 5×5pixels. Namely, the effect is great. Due to the fast processing of thelarge size filter, the following problems are solved.

2.7.1.1 Fast Processing of Obscure Image

As explained above, it is understood that processing of thetwo-dimensional filter having an effective size of 13×13 pixels can beeffectively performed with the capability of the conventional hardware.This size is about the same as that of a receptive field, and thereforea line segment can be extracted from an extremely obscure image whichcovers a receptive field. Thus, the problem of the fast processing ofthe obscure image is solved.

2.7.1.2 Fine Filtering of Line Segment

Although precise sorting and extraction of three types of line segments(line, edge, and gap) are required, this requirement is satisfied by analgorithm which is based on the fast processing with the one-dimensionalfilter, and is explained later. The above two effects are just anexample of the present invention, and the following new filter can berealized based on the speed-up due to the feature that "the function ofa two-dimensional filter can be performed by a one-dimensional filter".In addition, the "one-dimensional filter method" can exist in the sameway as above in an arbitrary input projective plane and an arbitrarytype of polar transformation. Namely, the present invention can benefitmany applications.

2.7.2 Filtering Impossible with Two-Dimensional Filter (Odd-FunctionFilter)

In addition to the speed-up of processing, filtering which is impossiblewith the conventional two-dimensional filter is made possible with useof the one-dimensional filter.

2.7.2.1 Projection of Two-Dimensional Filter Generates One-DimensionalEven Function Filter

There is a requirement for the two-dimensional filter that "the filterfunction per se must be isotropic". This is because, when the filterfunction is unisotropic, only a line segment in a specific direction isemphasized, and the extracted image becomes unnatural. Due to therequirement of isotropy, the two-dimensional filter is limited to be a"g(r)-type filter which depends on the distance r from the center only".Although the emphasis of a contour by using a two-dimensional filterhaving a diameter of about 30 elements is performed in the retina andthe lateral geniculate nucleus in the mammalian, the two-dimensionalfilter is an isotropic g(r)-type filter.

When the g(r)-type filter is made into a one-dimensional filter asexplained in the item of the principle of the one-dimensional filtermethod, the one-dimensional filter becomes an "even-function G(ρ) whichis symmetric about the center". Namely, the two-dimensional filterbecomes a one-dimensional even-function filter when the two-dimensionalfilter is projected, and an odd-function filter is not realized. Thus,the type of filters generated by projecting a two-dimensional filter islimited.

2.7.2.2 Arbitrary Function Filter is Possible by "Receptive FieldMethod+One-Dimensional Filter"

On the other hand, according to the "receptive fieldmethod+one-dimensional filter" in the present invention, isotropicfiltering is possible with an arbitrary function including theodd-function. The essential point is the application of aone-dimensional filter after being "mapped to all directions (polartransformation)". Since the isotropic processing is already applied bythe projection, the application of the one-dimensional filter is notaffected by the isotropy, and this is the reason why an arbitrary filtercan be applied.

From the above explanation, the following fact is understood. Theexplanation that "the processing of a two-dimensional filter can beperformed by a one-dimensional filter" is not sufficient. That is, thestatement that "the processing of a one-dimensional filter can beperformed by a two-dimensional filter" is not necessarily true.Describing more precisely, "polar transformation+one-dimensional filter"which gives the same output as "two-dimensional filter+polartransformation" necessarily exists. However, the inverse is not true.That is, "polar transformation+two-dimensional filter" which gives thesame output as "one-dimensional filter+polar transformation" exists onlywhen the one-dimensional filter is an even-function type.

An arbitrary filter is possible in the processing of "polartransformation+one-dimensional filter", and the possible arbitraryfilter includes a filter which is impossible in the processing of"two-dimensional filter+polar transformation" (such as an odd-functionfilter, a skeleton filter, and the like).

2.7.2.3 Characteristic of Odd-Function (Even-Function) Realizes aDifferential Filter of Odd (Even) Order

The "one-dimensional odd-function filter", which is not possible by theconventional two-dimensional filter, and becomes possible according tothe present invention, is an important filter which is indispensable forextraction of an "edge".

There is the following relationship between the odd-even symmetry of afilter and a differential operation.

The one-dimensional filter has a differential operation of an odd orderwhen the one-dimensional filter is an odd-function type.

The one-dimensional filter has a differential operation of an even orderwhen the one-dimensional filter is an even-function type.

Considering that the filtering is performed by convolution, the reasonwhy the above relationship exists can be understood below.

2.7.2.3.1 In Case of Odd-Function ##EQU15## As indicated above, theconvolution becomes a form of a difference of the first order, and afirst differentiation is performed. When the function g(α) contains acharacteristic of an odd function of a higher order, the differentiationis performed about the center of the symmetry thereof. Summarizing theabove, it is understood that "a one-dimensional filter of anodd-function performs differentiation of an odd order".

When the filter is not a complete odd-function in the above, that is

    g(α)=-η(α)·g(-α),

the function to be integrated becomes

    {f(ρ-α)-η(α)·f(ρ+α)}g(α)

to generate an offset component. However, the differentiation of thefirst order is still performed. Therefore, the operation ofdifferentiation of the first order is performed when the filter functionis not a complete odd-function as long as the function contains anodd-function component.

2.7.2.3.2 In Case of Even-Function ##EQU16## When the integration isreplaced with a difference in the above equation, and defining g_(a)(0)=g(0)/Na and Δa=1/Na, ##EQU17## Each term in the above sum ofproducts is sorted according to the state of the function g(α_(i)). Theterm in which g(α_(i)) is positive corresponds to an average operationas understood from the equation (b), and therefore the term functions asa low-pass filter. The term in which g(α_(i)) is negative corresponds toan operation of differentiation of the second order since the contentsof the parentheses { } in the equation (c) are in the form of adifference of the first order, and the difference operates in twostages. When g(α) contains a characteristic of an even-function of ahigher order, the above differentiation and the low-pass filter operatesabout the center of symmetry thereof. Summarizing the above, it isunderstood that "a one-dimensional filter of an even-function performsdifferentiation of the even order". The above low-pass filter can beconsidered as a differentiation of the order zero. Similar to the abovecase of the odd-function, when the function form of the filter is not acomplete even-function, the filter has a function of differentiation ofeven order as long as the filter has the following characteristic of theeven-function.

    g(α)=η(α)·g(-α)

From the above consideration, it is understood that the two-dimensionalfilters generally used are equivalent to even-function one-dimensionalfilters, and therefore the operations thereof are limited todifferentiation of an even order, and the differentiation of an oddorder, which is necessary for extraction of an "edge", cannot beperformed by the two-dimensional filters.

2.7.2.4 Odd-function Filter is Necessary for Extract of Edge

The characteristic which most frequently appears in an input image, andwhich is most credible, is a border between a luminous portion and adark portion. This border is called an edge. For extracting the border,an operation of extracting a portion at which variation of brightness ismaximum, is necessary, and mathematically, spatial differentiation of anodd order must be performed. FIGS. 36A, 36B, and 36C are output responsediagrams for a case when a one-dimensional first differential filter isapplied to an input image of an edge, where the output responseindicates a peak at the center of the edge, and therefore an edge can beeasily extracted.

On the other hand, according to the conventional two-dimensional filter,an odd-function filter cannot be applied, and therefore the edge cannotbe extracted. Therefore, conventionally, an indirect method is used.That is, a two-dimensional Gaussian filter (even-function) is applied toan input image, and differentiation of the second order is performed.Due to the differentiation of the second order, the output responsebecomes zero at the center (border) thereof where the variation ofbrightness is maximum. Therefore, the "zero-cross point" is extracted asan "edge" through troublesome processing. FIGS. 36D, 36E, and 36F areoutput response diagrams for a case when a two-dimensional seconddifferential filter is applied to an input image of an edge, and theoutput response is equal to zero at the center of the edge. However,since the "portion of zero" is weak against noise, it is difficult toprecisely extract the edge.

As explained above, according to the one-dimensional filter method inthe present invention, the differentiation of an odd order is possible,and therefore a maximum output is obtained at the "border", and theextraction of an edge can be stably performed. Further, the primaryvisual cortex in the cerebrum of a mammalian contains cells(hypercolumn) for extracting an edge, are regularly arranged in rows,and a maximum response is obtained at the location of an edge. Thisindicates that a filter for differentiation of an odd order is realizedtherein. This operation of the primary visual cortex in the cerebrum ofa mammalian can be modeled, and the flow of the operation is expressedas "two-dimensional second differential filter→polartransformation→first differential filter", which realizes a thirddifferential filter as a whole to enable the extraction of an edge.

2.7.2.5 Result of Simulation

A result of a simulation, wherein the above model of the visualinformation processing by the mammalian is simulated for a real image,is indicated in FIGS. 37A, 37B, 37C, and 37D. FIGS. 37A and 37B are acontour map and a schematic diagram of an edge as a real image,respectively. FIGS. 37C and 37D indicate a response (brightness) of thehypercolumn by contour lines, FIG. 37C indicates a response (brightness)of the hypercolumn by a contour map, and FIG. 37D indicates across-section of the contour map. The edge EDG in the round receptivefield CRC is extracted as a prominent peak PK on the hypercolumn planeof FIGS. 37C and 37D, and it is understood that the above odd-functionfilter is necessary to extract an "edge".

For comparison, a result of a simulation by the conventionaltwo-dimensional filter method "the two-dimensional second differentialfilter+the polar transformation", is indicated in FIGS. 38A and 38B. Atwin-peak output without a main peak appears in FIGS. 38A and 38B,wherein the output is zero at the location of an edge.

In the above simulation, a two-dimensional second differential filter isapplied first for comparison with the visual information processing inthe mammalian. However, this can be replaced with application of aone-dimensional second differential filter after the polartransformation to obtain the same output as above. Similar to the caseexplained before in the items of the speed-up, the amount of processingof filtering is greatly reduced by the replacement. In the visual cortexof a mammalian it seems that the one-dimensional filtering after thepolar transformation, which minimizes the amount of wiring, is notperformed in order to be commonly used for control of eyeballs andpreprocessing of colors, the outputs in which contours are enhanced.

2.7.3 Filtering Process Impossible by Two-Dimensional Filter (SkeletonFilter)

2.7.3.1 Skeleton Filter

According to the one-dimensional filter method, a filtering processwhich is impossible by the conventional two-dimensional filter becomespossible by using a filter other than the above odd-function filter. Thefilter is called a skeleton filter. As indicated in FIGS. 39A and 39B,the skeleton filter is a filter represented by Dirac's δ-functions. FIG.39A indicates a skeleton-type one-dimensional first differential filter,and FIG. 39B indicates a skeleton-type second differential filter. Thevalues of the skeleton-type one-dimensional first differential filter ofFIG. 39A, at the portions which are indicated by solid lines, are δ(0)and δ(0), respectively; and the values of the skeleton-typeone-dimensional second differential filter of FIG. 39B, at the portionswhich are indicated by solid lines, are -δ(0), +2δ(0), and -δ(0),respectively. The values of the skeleton-type one-dimensional firstdifferential filter and the skeleton-type one-dimensional seconddifferential filter, at the portions other than the above portionscorresponding to the solid lines, are zero. The values of theskeleton-type one-dimensional first differential filter, are -1, and 1,at the positions indicated by solid lines in FIG. 39A, respectively. Thevalues of the skeleton-type one-dimensional second differential filter,are -1, 2, and -1, at the positions indicated by solid lines in FIG.39B, respectively.

When the width of the filter is equal to 2a, the skeleton-typeone-dimensional first differential filter is expressed as

    G(ρ)=δ(ρ-a)-δ(ρ+a),

and the skeleton-type second differential filter is expressed as

    G(ρ)=2δ(ρ)-δ(ρ-a)-δ(ρ+a).

2.7.3.2 Convolution Calculation by Skeleton Filter

According to the skeleton filter, the integration in the convolutioncalculation becomes a simple sum of products. When the polar-transformedim data is expressed by F(ρ), in the case of the skeleton-typeone-dimensional first differential filter, the convolution output C(ρ)is expressed as ##EQU18## In the case of the skeleton-type seconddifferential filter, the convolution output C(ρ) is expressed as##EQU19## the integration can be expressed by a simple combination ofthe input data F(ρ) to greatly reduce the amount of processing.

2.7.3.3 Embodiment of Skeleton Filter

A line segment can be precisely extracted by such a simple skeletonfilter. FIG. 40 is a diagram of a construction wherein theone-dimensional filter is constituted by a skeleton filter. In FIG. 40:reference numeral 31 denotes an input memory for storing an image of anobject, having a size of N×N pixels; 32 denotes a receptive fielddivision portion for dividing the input image into receptive fields; 33denotes a polar transformation portion for applying predetermined polartransformation to the receptive field image; 34 denotes aone-dimensional filter comprised of a skeleton-type first differentialfilter 34a and a skeleton-type second differential filter 34b; 35denotes a hypercolumn memory for storing hypercolumn images; and 36denotes a characteristic feature extracting portion.

The skeleton-type first differential filter 34a has a characteristicexpressed as

    G(ρ)=δ(ρ-1)-δ(ρ+1),

and the skeleton-type second differential filter 34b has acharacteristic expressed as

    G(ρ)=2δ(ρ)-δ(ρ-2)-δ(ρ+2).

A result of extraction of an "edge of a line segment" by the skeletonfilter of FIG. 40, is indicated in FIGS. 41A, 41B, 41C and 41D. FIG. 41Ais a contour map of an edge as a real image, where the hatched portioncorresponds to a dark portion, and the portion not hatched correspondsto a bright portion. FIG. 41B indicates a response (brightness) of thehypercolumn by contour lines. The edge EDG in the round receptive fieldCRC is extracted as a prominent peak PK on the hypercolumn plane of FIG.41B. Thus, it is understood that the function of extracting an edge issufficiently realized by the skeleton filter. FIGS. 41C and 41D arecross-sectional views of the contour maps of FIGS. 41A and 41B,respectively.

2.7.3.4 Skeleton Filter as Basic Function of Differentiation

The reason why the skeleton filter has a sufficient function of the edgeextraction, is considered below. For example, a first differentialfilter can be expressed as below,

    H(ρ)=Γ(ρ-a)-Γ(ρ+a)

where Γ(ρ) is a mountain-shaped function, and the function is deformedas follows.

    =∫Γ(ρ-ξ){δ(ξ-a)-δ(ξ+a)}dξ

    =∫Γ(ρ-ξ)G(ξ)dξ

Namely, the usual filter is a "filter generated by synthesizing askeleton filter G(ρ) with a low-pass filter Γ(ρ)". Analyzing thefunction thereof, the operation of the first differentiation, which isthe object, is performed by the skeleton-type first differential filterG(ρ), and Γ(ρ) has an auxiliary function to reduce a high frequencynoise.

Thus, the skeleton filter has a basic function of the differentialoperation, and is a basic filter for extracting an edge. This is thereason why the skeleton filter has a sufficient function of the edgeextraction.

2.7.3.5 Advantage of Skeleton Filter

The width of a mountain-shaped function Γ(ρ) for a filter usually usedis 3 to 5 pixels, and the amount of processing is larger than that ofthe skeleton-type filter, by the multiple corresponding to the width.Therefore, the amount of processing is reduced by a factor 3 to 5 by useof the skeleton filter.

2.7.3.6 Skeleton Filter is Impossible to Realize by ConventionalTwo-Dimensional Filter

The one-dimensional skeleton filter cannot be made by a two-dimensionalfilter. The reason is explained below.

Due to the requirement for isotropy as explained before, thetwo-dimensional filter becomes a concentric circular filter, and isconstituted by a combination of ring filters having a zero width. Thiselement ring filter is the sharpest filter in two dimensions. However, askeleton filter, i.e., a filter which is equal to zero at locationsother than the filter point, cannot be realized by the element ringfilter even when the element ring filter is mapped into one dimension,because the ring is continuous, and a discontinuous filter which isequal to zero at locations other than the filter point, cannot begenerated by projection. Therefore, there is no two-dimensional filterwhich requires as small a processing amount as that required by theskeleton filter.

2.7.3.7 Summary

The above explanations are summarized as follows.

The skeleton filter is possible only for "polartransforation+one-dimensional filter" (which is expressed by Dirac's δfunctions).

The skeleton filter can reduce the amount of convolution calculation bya factor 1/3˜1/5 compared with the two-dimensional filter.

The convolution calculation of the sharpest two-dimensional filter(having a zero width ring) can be reduced to calculation of three pixelsof the center pixel and pixels at both ends. The reduced amount ofprocessing is 3/(ρN_(r)) when the diameter of the ring filter is N_(r),and contribution to the speed-up is great. For example, N_(r) ≧6 isrequired for processing an obscure image, and the amount of processingis reduced by a factor of 1/2ρ.

2.7.4 Multi-filter

Since, according to "receptive field method+one-dimensional filter" inthe present invention, the operation of a large mask can be performedwith a high speed as described above, simultaneous application of aplurality of filters respectively having different widths at the sametime (multi-filter) is possible. Thus, "simultaneous extraction of sharpportions and obscure portions in an image", which is impossible by theconventional method, can be performed, and working and moving operationsbased on recognition of the whole image becomes possible. In the visualcortex of a mammalian, simultaneous application of a plurality of typesof filtering respectively corresponding to different widths areperformed to ensure the recognition of characteristic features.

FIG. 42 is a diagram illustrating a construction of thethree-dimensional measurement system using a multi-filter. In FIG. 42:reference numeral 41 denotes an input memory for storing an input imageof N×N; 42 denotes a receptive field division portion for dividing theinput image into receptive fields to output the divided image; 43denotes a polar transformation portion for performing predeterminedpolar transformation on the receptive field images; 44 denotes aone-dimensional multi-filter comprising: a one-dimensional filter 44awith a width W₁, a one-dimensional filter 44b with a width W₂, aone-dimensional filter 44c with a width W₃, a one-dimensional filter 44dwith a width W₄, . . . , and a synthesis portion 44e for synthesizingoutputs of the one-dimensional filters to output the synthesized output;45 denotes a hypercolumn memory for storing a hypercolumn image; and 46denotes a characteristic feature extraction portion.

In this system, an input image stored in the input memory 41 is dividedinto receptive fields by the receptive field division portion 42, andthe polar transformation portion 43 is performed on the receptive fieldimages. Next, the one-dimensional filter processing in the ρ-directionin the hypercolumn plane, including a plurality of types of filteringwith a plurality of different filter widths in parallel, is performed onthe polar-transformed output. Then, outputs of the plurality ofone-dimensional filtering are synthesized to obtain a one-dimensionalmulti-filter output, and lines, edges, and gaps are extracted by thecharacteristic feature extraction portion 46 based on theone-dimensional multi-filter output. By the above construction, it ispossible to "extract simultaneously sharp to obscure line segments" bysynthesizing the multi-filter with widths W₁ ˜W₄, based on themulti-filter output. This is one of the techniques which is firstrealized by the high speed filtering according to the present invention.

Although the widths W₁ ˜W₄ in the above multi-filter may be arbitrary, amulti-filter having an exponential form (normally, n=2 is sufficient) asdescribed below is preferable since it is not necessary to provide finewidth steps for relatively larger widths considering efficiency inprocessing.

    W.sub.i =(W.sub.i-1).sup.n

The "two-dimensional filter+receptive field method" using theconventional two-dimensional filter, has the following problems whichmake realization of the multi-filter impossible.

The amount of processing increases greatly. When the number of filtersin the multi-filter is μ, the amount of processing increases by a factorμ×(m/2), compared with the one-dimensional filter method according tothe present invention, and therefore the multi-filter cannot be usedfrom the view point of engineering.

The differentiation of an odd order is impossible. The two-dimensionalconvolution filter is limited to a ring filter, which cannot perform thedifferentiation of an odd order, and cannot detect an "edge". Therefore,it is impossible to form a multi-filter by the conventionaltwo-dimensional filter.

2.7.5 Positive-Negative-Separation-Type One-Dimensional Multi-StageFiltering

The above-mentioned edge extraction filter, i.e., a filter with aconstruction of "division into receptive fields→polartransformation→one-dimensional differential filter→second differentialfilter". However, this type of edge extraction filter still has aproblem. This is because the extraction of an edge becomes impossibledue to interference of outputs from borders on both sides of a luminous"band" when a width of the luminous "band" is small. The cause is that,as indicated in FIG. 43, positive and negative peaks corresponding toborders on both sides of the band, output from the first differentialfilter 2, come close together when the width of the band becomes small,and interference occurs due to further second differentiation. In FIG.43, reference numeral 1 denotes an image of an edge, 2 denotes a result(first differentiation output) of the first differentiation applied onthe image of the edge, and 3 denotes a second differentiation output.

To suppress the interference, the positive and negative peaks in thefirst differentiation output are separated, and second differentiationis performed independently. FIG. 44 is a diagram illustrating theconstruction of a positive-negative-separation-type one-dimensionalmulti-stage filter. In FIG. 44: reference numeral 52 denotes a receptivefield division portion for dividing an input image; 53 denotes a polartransformation portion for performing a predetermined polartransformation on the receptive field images; 54 denotes apositive-negative-separation-type one-dimensional multi-stage filtercomprising a one-dimensional differential filter 54a, apositive-negative separation circuit 54b for separating positive andnegative signals in the output of the one-dimensional differentialfilter, first and second differential filters 54c and 54d for applyingsecond differential filtering to the positive and negative portions ofthe first differentiation output, respectively, a positive selectingportion 54e for selecting a positive signal from the output of thesecond differential filter 54c, a negative output selecting portion 54ffor selecting a negative signal from the output of the seconddifferential filter 54d, and a synthesis portion 54g for synthesizingthe outputs of the positive and negative output selecting portions.

FIG. 45 indicates a result of a simulation of thepositive-negative-separation-type one-dimensional multi-stage filter. InFIG. 45, reference numeral 1 denotes an image of an edge, 2 denotes apositive output (first differentiation output) when the firstdifferentiation is applied to the edge image, and 3 denotes an output(the output of the second differential filter 54c) when the seconddifferentiation is applied to the positive first differentiation output.Compared with the case wherein positive-negative separation is notperformed, interference of the peaks output corresponding to the edgesis effectively suppressed. In the cerebrum, almost all processing isperformed with positive and negative outputs being separated. This isbecause neurons cannot transmit a negative signal, and therefore thenegative output components are transmitted through wiring (axon) fornegative signals, provided in parallel to wiring for positive signals.According to this construction in the cerebrum due to the limitation inthe neuronal system, interference can be suppressed.

FIG. 46 is a diagram illustrating the construction of apositive-negative-separation-type multi-filter in the case where theabove positive-negative separation system is adopted to themulti-filter. In FIG. 46: reference numeral 62 denotes a receptive fielddivision portion for dividing an input image into receptive fields, andoutputting the divided image; 63 denotes a polar transformation portionfor performing predetermined polar transformation on the receptive fieldimage; 64 denotes a positive-negative-separation-type multi-filtercomprising a one-dimensional filter 64a-1 with a width W₁₁, aone-dimensional filter 64a-2 with a width W₁₂, a one-dimensional filter64a-3 with a width W₁₃, a one-dimensional filter 64a-4 with a width W₁₄,positive-negative separation circuits 64b-1˜64b-4 for separatingpositive and negative signals in the outputs of the respectiveone-dimensional differential filters, first and second differentialfilters 64c-1, 64c-2; 64d-1, 64d-2; 64e-1, 64e-2; 64f-1, and 64f-2(respectively having widths W₂₁, W₂₂, W₂₃, W₂₄) for applying seconddifferential filtering to the positive and negative portions of thefirst differentiation output, respectively and independently, asynthesis portion 64g for synthesizing positive signals from the seconddifferential filters, and outputting the synthesized signal, and asynthesis portion 64h for synthesizing negative signals from the seconddifferential filters, and outputting the synthesized signal.

In the above positive-negative-separation-type multi-filter, positiveand negative signals in the output of each first differential filter areseparated, and the second differential filtering is appliedindependently to each of the positive and negative signals. Then, thepositive signals and negative signals are respectively synthesized.Thus, precise extraction of obscure portions and narrow bands in animage, suppressing interference, can be performed.

The simulation result of the positive-negative-separation-typemulti-filter is indicated in FIGS. 47A, 47B and 48. FIG. 47B indicates ahypercolumn image by a contour map, where the hypercolumn image isobtained by applying the positive-negative-separation typemulti-filtering to a receptive field image (the magnification of whichis indicated in FIG. 47A) in the encircled area of the original image(FIG. 13). Four edges in the receptive field are extracted as four sharppeaks P1˜P4. FIG. 48 indicates a regenerated image obtained byextracting peaks as above in all the hypercolumns to regenerate linesegments based on an orientation, a location, and a length thereof. Dueto the multi-filter and the positive and negative separation, obscureportions and narrow bands existing in the original image are extractedstably.

In the above simulation, the widths W₁₁ ˜W₁₄ of the first differentialfilters 64a-1˜64a-4 are set as equal to 1, 2, 4, and 8 pixels,respectively; the widths W₂₁, W₂₂, W₂₃ and W₂₄ of the seconddifferential filters 64c-1, 64c-2; 64d-1, 64d-2; 64e-1, 64e-2; 64f-1,and 64f-2 are set as equal to 2, 2, 4, and 8 pixels, respectively; and asum of the outputs of the synthesis portions 64g and 64h is indicated inthe above figures.

As understood from the above explanation and simulation, according to"receptive field method+polar transformation+positive and negativeseparate processing", characteristic features such as narrow bands, highdensity stripes, and the like, can be extracted by suppressinginterference. Not limited to the extraction of edges, the aboveseparation process can be applied to general processing whilesuppressing interference.

2.7.6 One-Dimensional Filter Regarding Time

In the above explanation, a one-dimensional filter is applied spatiallyon the dual plane (the hypercolumn plane), and a one-dimensional filtermay be operated regarding time by obtaining the difference between animage (image frame) at a timing and an image at a next timing. Thereby,characteristic feature amounts of a moving object, as explained below,can be recognized by using the one-dimensional filter for which theamount of processing is small.

When the filter is operated in the ρ-direction regarding time,characteristic quantities (movement direction and movement amount, andthe like) of an object translating in the receptive field, can beextracted by one-dimensional processing.

When the filter is operating in the θ-direction regarding time,characteristic quantities (rotation direction and rotation amount, andthe like) of an object rotating in the receptive field, can be extractedby one-dimensional processing.

In addition, due to the characteristic of the convolution integrationfilter, the one-dimensional filter is resistant to noise.

In the hypercolumn in the primary visual cortex of the cerebrum, cells(complex cell) for performing an operation of the one-dimensionaldifferential filter regarding time, are provided to extractcharacteristic features of a moving object only. In addition, there is afeedback from the fifth layer in the hypercolumn to the superiorcolliculus which controls steady gaze operations, and movement directionand movement velocity are transmitted.

Although, up to now, extraction of the movement characteristic featureshas been tried, for example, by the gradient-type optical flow method,the optical flow method is not resistant to noise since a differentialfilter is used therein. On the other hand, according to the aboveone-dimensional filter regarding time, stable extraction is possiblesince the filter is an integration-type.

2.8 Embodiment of Each Type of Filter

2.8.1 Contour Line Extraction Filter

Contour lines are the most basic characteristic feature in images. Theenvironment and almost all the working objects in industrial plants andfactories are constructed in the form of straight lines and cylindersfor ease in manufacturing, and these straight lines and cylinders areprojected in an image (screen) as straight lines. Further, the remainingcurves in the image can be approximated as straight lines when viewedfrom a short range, and these straight lines and cylinders can beextracted as a group of tangential lines (envelope curves).

Namely, the above tangential lines are detected by the receptive fieldmethod wherein the image is processed after being divided into smallareas. Therefore, by the receptive field method, "extraction of anenvelope curve of a curve" is possible, and almost all characteristicfeatures except very small figures, can be extracted. In imageprocessing, extraction of contour lines (tangential lines) by thereceptive field method plays a basic role. In the primary visual cortexof the cerebrum (hypercolumn), the tangential line extraction by thereceptive field division is performed first since it is importantpreprocessing.

The contour lines are extracted from three types of states of theoriginal image, and filtering suitable for the respective types ofstates must be applied to the image. The three types of contour linesare a line, an edge, and a gap, where the "line" is a contour linesegment in the form of a narrow band, the "edge" is a contour linesegment in the form of a border line between a luminous portion and adark portion, the "gap" corresponds to a "line" when the brightness ofthe line is inverted. Concrete filters suitable for extraction of therespective types, are explained below.

For simplicity of the explanations, abbreviations for various types ofone-dimensional and two-dimensional filters are defined as follows.FIGS. 49A, 49B, and 49C are diagrams for indicating filter symbols,gr_(a) (or gr), gas_(a) (or gas), and 2gas_(a) (or 2gas). Forone-dimensional filter, a skeleton-type first differential filter havinga width a is indicated by gr_(a) or gr (see FIG. 49A), a skeleton-typesecond differential filter having a width a is indicated by gas_(a) orgas (see FIG. 49B). For two-dimensional filters, a second differentialfilter having a diameter a is indicated by 2gas_(a) or 2gas (see FIG.49C). In the two-dimensional filter, no odd-function filter exists, andtherefore no first differential filter can be defined. In thespecification, the one-dimensional gr filter and the one-dimensional gasfilter are not limited to the skeleton-type, and include amountain-shaped gradient filter and a Gaussian filter.

2.8.2 "Line" Extraction Filter

FIGS. 50A, 50B and 50C are diagrams illustrating various constructionsof "line" extraction filters, FIG. 50A is a diagram illustrating aconstruction of a basic filter which extracts a "line" by the receptivefield method (receptive field division+polar transformation) only, FIG.50B is a diagram illustrating a construction of a line extraction filterby the process of "receptive field method+one-dimensional gas filter",and FIG. 50C is a diagram illustrating a construction of a lineextraction filter by "two-dimensional gas filter+receptive field".

2.8.2.1 The line extraction filter of FIG. 50A is a basic filter forline extraction, wherein an input image is divided into receptive fieldsby the receptive field division portion 71, and polar transformation isapplied to each receptive field image by the polar transformationportion 72 to extract a line. The "line" can be extracted by using theoutput of the receptive field method only. However, when extracting a"line" by using the output of the receptive field method only, there isa drawback that outputs are also great in the other portions which donot correspond to the "line" and have uniform brightness, and thereforethe filters of FIGS. 50B and 50C are desirable, where a positive peakcorresponds to a "line", and the bottom of a valley corresponds to a"gap" in the output of the basic filter.

A simulation result wherein the "line" is extracted by using the basicfilter is indicated in FIGS. 51A and 51B. Since only a "line" SL existsin the round receptive field CRC (FIG. 51A), the above drawback does notappear, and the line is extracted as a positive peak PK on thehypercolumn (FIG. 51B).

2.8.2.2 In the line extraction filter of FIG. 50B, an input image isdivided into receptive fields by the receptive field division portion71, polar transformation is applied to each receptive field image by thepolar transformation portion 72, and one-dimensional second differentialfilter processing is applied to the polar transformation output by theone-dimensional gas filter 73 to extract a line segment. According tothe line extraction filter, a main peak corresponding to a "line" isobtained together with sub-peaks having a polarity opposite to the mainpeak, on both sides of the main peak. The sub-peaks play an importantrole to emphasize the main peak in the processing following thisoperation. The above portions of the uniform brightness are eliminatedthrough the differentiation by the filter. The positive peak correspondsto a "line". The amount of processing is small since the filter is aone-dimensional filter, and the filter is the most superior "line"extraction filter.

When the width of the gas filter is too small, a "line" may be deemed asa band having a width, so differentiation is performed on edges on bothends of the band, and therefore the output becomes zero at the mainpeak. To avoid this problem, the following condition is required.

The diameter (a) of the gas filter≧the width of the "line".

The optimum condition is the diameter (a) of the gas filter=twice thewidth of the "line".

A simulation result of the "line" extraction under optimum conditions isindicated in FIGS. 30A and 30B. A line corresponding to the positivemain peak PK is detected accompanied by the negative sub-peaks PK1 andPK2.

2.8.2.3 In the line extraction filter of FIG. 50C, the two-dimensionalsecond differentiation processing is first applied to an input image bythe two-dimensional gas filter 74, then division into receptive fieldsis performed, and polar transformation is applied to each receptivefield image to extract a line.

The above line extraction filter is equivalent to the filter of FIG.50B. A peak corresponding to a "line" is obtained accompanied bysub-peaks having a height equal to half of the height of the main peak,and a polarity opposite to that of the main peak, on both sides of themain peak. However, as mentioned above, the amount of processing of thefilter of FIG. 50B is much smaller than the filter of FIG. 50C. Thepositive peak corresponds to a line.

When the width of the 2gas filter is too small, a "line" may be deemedas a band having a width, so differentiation is performed on edges onboth ends of the band, and therefore the output becomes zero at the mainpeak. To avoid this problem, the following condition is required.

The diameter (a) of the 2gas filter≧the width of the "line".

The optimum condition is the diameter (a) of the 2gas filter=twice thewidth of "line".

A simulation result of the "line" extraction under optimum conditions isindicated in FIGS. 29A and 29B. A line corresponding to the positivemain peak PK is detected accompanied by the negative sub-peaks PK1 andPK2. Namely, the same output as the line extraction filter of FIG. 50Bis obtained.

2.8.3 Edge Extraction Filter

First, the principle and the advantage of the "edge" filter areexplained. Among the three types of line segments, the "edge" appearsmost frequently. The reason is because the numbers of the "lines" and"gaps" are relatively small since these are fine characteristicfeatures, and because almost all characteristic features appearing inwide views of images are the "borders at which brightness varies", i.e.,"edges". However, since filters of an odd order are not allowed as theconventional two-dimensional filter, the "edge" cannot be extracted.

On the other hand, an arbitrary filter is allowed according to theone-dimensional filter method, an "edge" can be extracted through thefollowing basic flow. The hypercolumn in the primary visual cortex ofthe cerebrum contains cells for extracting the above important "edge".As explained in detail below, the basic flow of the "edge" extraction is"receptive field division→polar transformation→first differentialfilter→second differential filter", and the function is as follows.Namely,

Function of First Differentiation Filter: the first differential filtertransforms variations in brightness in an error function-like (bluntstep-like) shape, to a mountain-shaped output at which a ratio ofvariation is maximum. Thus, the first differential filter is a filterwhich transforms an "edge" to a "line", and this is a basic filter for"edge" extraction. An "edge" can be extracted by this filter only.

Function of Second Differentiation Filter: a "line" which is transformedby the first differential filter is made sharp by the line extractionfunction of the second differential filter, and sub-peaks having apolarity opposite to the main peak, at both ends of the "line" areobtained. The function of the second differential filter is importantfor making the main peak prominent in the following processing, and isexactly the function of the aforementioned line extraction filter.

Based on the above explanation, the basic flow of the "edge" extractioncan be expressed as, "receptive field division→polartransformation→"line" transformation filter→"line" extraction filter".Unless a non-linear operation is added, the order of the filter can bechanged to an arbitrary order. Various types of "edge" filters areexplained below.

FIGS. 52A, 52B, 52C, and 52D are diagrams illustrating constructions ofthe various types of "edge" extraction filters. FIG. 52A is a diagramillustrating the construction of the basic filter wherein theone-dimensional gr filter processing is applied to the output of thereceptive field method (receptive field division+polar transformation)to extract the "edge", FIG. 52B is a diagram illustrating theconstruction of the edge extraction filter wherein the one-dimensionalgas filter is connected in the stage following the construction of FIG.52A, and FIG. 52C is a diagram illustrating the construction of the edgeextraction filter wherein the two-dimensional gas filter is connected inthe stage preceding the construction of FIG. 52A, and FIG. 52D is adiagram illustrating the construction of the edge extraction filterwherein the one-dimensional gr filter in the construction of FIG. 52C isreplaced with a one-dimensional gr filter regarding time.

2.8.3.1 In the edge extraction filter of FIG. 52A, an input image isdivided into receptive fields by the receptive field division portion81, polar transformation is applied to each receptive field image by thepolar transformation portion 82, and first differentiation processing isapplied to the output of the polar transformation by the one-dimensionalgr filter 83 to extract an "edge", since the "edge" can be extracted bythe one-dimensional gr filter only. Namely the edge extraction filter ofFIG. 52A is a basic filter for the "edge" extraction. However, since themain peak output from the edge extraction filter of FIG. 52A is notaccompanied by the sub-peaks of a polarity opposite to the main peak,the processing following the operation cannot be performed sharply, andfilters explained with reference to FIG. 52B and later are preferable.

A simulation result of the "edge" extraction by using the basic filteris indicated in FIGS. 53A and 53B. The "edge" EDG in the round receptivefield CRC (FIG. 53A) is extracted on the hypercolumn, as a main peak PK(FIG. 53B) which is not accompanied by sub-peaks.

2.8.3.2 In the edge extraction filter of FIG. 52B, an input image isdivided into receptive fields by the receptive field division portion81, polar transformation is applied to each receptive field image by thepolar transformation portion 82, first differentiation processing isapplied to the output of the polar transformation by the one-dimensionalgr filter 83, and further one-dimensional second differentiationprocessing is applied to extract an "edge".

Due to the function of sharpening of the one-dimensional gas filter 84,the main peak corresponding to an "edge" is obtained accompanied bysub-peaks having a height equal to half of the height of the main peak,and a polarity opposite to that of the main peak, on both sides of themain peak. The sub-peaks play an important role to emphasize the mainpeak in the processing following this operation. The amount ofprocessing is small since the edge extraction filter of FIG. 52B is aone-dimensional filter. Namely, the edge extraction filter of FIG. 52Bis the most superior "edge" filter.

When the width of the gas filter is too small, the "line" transformed bythe one-dimensional gr filter may be deemed as a band having a width, sodifferentiation is performed on edges on both ends of the band, andtherefore the output becomes zero at the main peak. To avoid thisproblem, the following condition is required.

The diameter (a) of the gas filter≧the width (b) of the gr filter.

The optimum condition is the diameter (a) of the gas filter=twice thewidth (b) of the gr filter.

A simulation result of the "edge" extraction under optimum conditions isindicated in FIGS. 54A and 54B. The positive peak corresponding to the"edge" EDG in the round receptive field CRC (FIG. 54A) is extractedaccompanied by negative sub-peaks PK1 and PK2 (FIG. 54B).

2.8.3.3 In the edge extraction filter of FIG. 52C, first, thetwo-dimensional gas filter 85 applies two-dimensional seconddifferentiation processing to an input image, then the processed imageis divided into receptive fields by the receptive field division portion81, polar transformation is applied to each receptive field image by thepolar transformation portion 82, and first differentiation processing isapplied to the output of the polar transformation by the one-dimensionalgr filter 83 to extract an "edge".

The edge extraction filter of FIG. 52C is equivalent to the edgeextraction filter of FIG. 52B, the main peak corresponding to an "edge"is obtained accompanied by sub-peaks having a height equal to half ofthe height of the main peak, and a polarity opposite to that of the mainpeak, on both sides of the main peak. However, the amount of processingof the edge extraction filter of FIG. 52B, is much smaller than that ofthe edge extraction filter of FIG. 52C for the reason explained above.

When the diameter of the two-dimensional gas filter (2gas) is too small,the operation of the two-dimensional gas filter (2gas) is equivalent todeem the "line" transformed by the one-dimensional gr filter, as a bandhaving a width, and to perform differentiation on edges on both ends ofthe band, and therefore the output becomes zero at the main peak. Toavoid this problem, the following condition is required.

The diameter (a) of the 2gas filter≧the width (b) of the gr filter.

The optimum condition is the diameter (a) of the 2gas filter=twice thewidth (b) of the gr filter.

A simulation result of the "edge" extraction in the optimum condition isindicated in FIGS. 37C and 37D. The positive peak corresponding to theedge is extracted accompanied by negative sub-peaks PK1 and PK2.

2.8.3.4 When replacing the one-dimensional gr filter in FIGS. 52A˜C withthe one-dimensional gr filter regarding time (see, for example, FIG.52D), the replaced filter does not respond to a figure standing still,and only a translating "edge" is extracted. Movement direction andmovement amount are extracted, and thus, gazing and pursuit of anobject, and avoiding an obstacle based on the whole movement pattern,are possible. The hypercolumn in the primary visual cortex of thecerebrum also contains cells having a function of "two-dimensional gasfilter→receptive field division→polar transformation→one-dimensionalfilter regarding time", to extract a moving "edge".

2.8.3.5 Filter for Extracting "Edge" as Positive Signal

FIG. 55 is a diagram illustrating various types of constructions forextracting an "edge" as a positive signal, where the constructionscorrespond to the edge extraction filters indicated in FIGS. 52A and52B, and the same elements bear the same reference numerals in FIGS.52A, 52B, and 55.

In the edge extraction filter indicated as method 1 in FIG. 55, anabsolute value circuit 87 is provided in the stage following theone-dimensional gr filter 83 in FIG. 52A, to extract an edge as apositive signal. In the edge extraction filter indicated as method 2 inFIG. 55, an absolute value circuit 87 is provided in the stage followingthe one-dimensional gr filter 83, and a positive separation circuit 88is provided in the stage following the one-dimensional gas filter 84 inFIG. 52B, to extract an edge as a positive signal. In the edgeextraction filter indicated as method 3 in FIG. 55, positive-negativeseparation circuit 89 is provided in the stage following theone-dimensional gr filter 83 in FIG. 52B, one-dimensional seconddifferentiation processing is applied to positive and negativecomponents in the output of the one-dimensional gr filter by theone-dimensional gas filters 84a and 84b, respectively and independently;positive and negative signals are selectively output through thepositive select portion 89a and the negative select portion 89b,respectively; the polarity of the negative signal is inverted by thepolarity inversion circuit 90; and the positive signals are synthesizedby the synthesis portion 91 to output a positive edge signal.

FIG. 56 is a diagram illustrating another construction of a filter whichextracts an "edge" as a positive signal, and corresponds to the edgeextraction filter indicated in FIG. 52C. An absolute value circuit 87 isprovided in the stage following the one-dimensional gr filter 83 in FIG.52C, to extract an edge as a positive signal. By extracting an edge as apositive signal, the output of the filter is not affected by line noiseand the sub-peaks having a polarity opposite to the main peak.

2.8.4 Gap Extraction Filter

The gap extraction filter is constructed by inverting the polarity inthe construction for the line extraction filter. Namely, a gapextraction filter is constructed by connecting a polarity inversioncircuit in the stage following each line extraction filter in FIGS.50A˜50C. FIGS. 57A, 57B and 57C are diagrams illustrating theconstruction of a gap extraction filter, 75 denotes a polarity inversioncircuit. In FIGS. 50A˜50C and 57A˜57C, the same reference numerals areaffixed to the same elements.

2.8.4.1 Gap Extraction Filter Having Construction of "Receptive FieldDivision→Polar Transformation→Polarity Inversion"

In the gap extraction filter of FIG. 57A, a polarity inversion circuit75 is provided in the stage following the line extraction filter in FIG.50A. The "gap" can be extracted from the output of the receptive fieldmethod only. However, there is a problem that the output of the filteris large in uniformly dark portions, and therefore gap extractionfilters as explained below are preferable. A positive peak in the outputcorresponds to a gap.

2.8.4.2 Gap Extraction Filter Having Construction of "Receptive FieldDivision→Polar Transformation→One-Dimensional gas Filter→PolarityInversion"

In the gap extraction filter of FIG. 57B, a polarity inversion circuit75 is connected in the stage following the construction of FIG. 50B. Bythe one-dimensional gas filter 73, a main peak corresponding to a "gap"is obtained accompanied by sub-peaks having a polarity opposite to themain peak, and a height equal to half of the height of the main peak, onboth sides. The sub-peaks play an important role to emphasize the mainpeak in the processing following this operation. The uniformly darkportions are eliminated by the differentiation operation. The amount ofprocessing is small since the filter is a one-dimensional filter, andthe filter is the most superior "gap" extraction filter.

When the diameter of the gas filter is too small, a "gap" may be deemedas a band having a width, so differentiation is performed on edges onboth ends of the band, and therefore the output becomes zero at the mainpeak. To avoid this problem, the following condition is required.

The diameter (a) of the gas filter≧the width of the "gap".

The optimum condition is the diameter (a) of the gas=twice the width of"gap".

2.8.4.3 Gap Extraction Filter Having Construction of "Two-Dimensionalgas Filter→Receptive Field Division→Polar Transformation→PolarityInversion"

In the gap extraction filter of FIG. 57C, a polarity inversion circuit75 is connected in the stage following the construction of FIG. 50C. Thegap extraction filter of FIG. 57C is equivalent to the gap extractionfilter of 2.8.4.2. A main peak corresponding to a "gap" is obtainedaccompanied by sub-peaks having a height equal to half of the height ofthe main peak, and a polarity opposite to that of the main peak, on bothsides of the main peak. However, as mentioned above, the amount ofprocessing of the filter of 2.8.4.2 is much smaller than the filter ofFIG. 57C.

When the diameter of the 2gas filter is too small, a "gap" may be deemedas a band having a width, differentiation is performed on edges on bothends of the band, and therefore the output becomes zero at the mainpeak. To avoid this problem, the following condition is required.

The diameter (a) of the 2gas filter≧the width of the "gap".

The optimum condition is the diameter (a) of the 2gas filter=twice thewidth of "gap".

2.8.5 Extraction Filter for "Line" and "Gap" Only

As explained above, although filters for extracting three types of linesegments are obtained, there is a problem that an "edge" is included inthe outputs of the "line" extraction filter and the "gap" extractionfilter. The reason is because, in the line extraction filter, a "gap" istransformed to a pair of positive and negative peaks, and therefore asignal corresponding to a "gap" cannot be discriminated from a signalcorresponding to a "line" based on the signal value only. FIG. 58 is adiagram illustrating a construction of an "only-line extraction filter"which eliminates noise corresponding to an "edge" to extract a "line"only. The "only-line extraction filter" is constructed based on the factthat the basic filter for extracting a "line" (FIG. 50A) does notrespond to an "edge".

In FIG. 58, reference numeral 101 denotes a basic filter constructed bya receptive field division portion 71 and a polar transformation portion72, 102 denotes a line extraction filter which is constructed byconnecting a one-dimensional gas filter 73 to the basic filter (see FIG.50B), 103 denotes a peak detect portion which detects a peak in theoutput of the basic filter, 104 denotes a line emphasis portioncomprised as a gate for passing therethrough an output portion in thevicinity of a peak which is detected by the peak detection portion 103,in the output of the line extraction filter 102.

The basic filter 101 is a prototype filter for line extraction. Sincethe basic filter 101 does not comprise a first differential filter, whena variation in brightness is small, a portion at which the variation inbrightness is maximum, is extracted as a peak, while no peak isextracted at a portion of an "edge". On the other hand, the lineextraction filter 102 suppresses a portion where the variation ofbrightness is small by the first differential filter. However, "edge"noise is generated by the line extraction filter 102 as explained above.When a peak is detected from the output of the basic filter 101 by thepeak detection portion 103, this peak is due to a "line" other than anedge. Therefore, only a portion in the vicinity of the peak of theoutput of the line extraction filter 102 passes through the lineemphasis portion 104, and the other portion of the output of the lineextraction filter 102 is stopped. Thus, a signal of "line only" isoutput.

Instead of the above "gate" construction, the line emphasis portion 104may be constructed so that a peak detection output is multiplied by theoutput of the line extraction filter, or is added to the output of theline extraction filter, to emphasize a line portion.

In addition, since "only-gap" extraction is performed by obtaining acomplementary signal of a "line" signal, processing similar to the abovemay be performed with inverting the polarity.

2.8.6 "Only-Edge" Extraction Filter

Although, by the edge extraction filter, a narrow band can be extractedby extracting both ends of the narrow band as "edges", signalsrepresenting the "edges" corresponding to both ends interfere with eachother to lower the credibility, when the band is too narrow. Therefore,it is natural to extract such a narrow band by the above only-lineextraction filter, and therefore it is desirable to eliminate thesignals representing the "edges" from the output.

FIG. 59 is a diagram illustrating the principle of the only-edgeextraction filter. In FIG. 59, reference numeral 111 denotes anonly-line extraction filter, 112 denotes an only-edge extraction filter,113 denotes a line elimination portion which subtracts the output of theonly-line extraction filter from the output of the only-edge extractionfilter, to output an "edge only" signal. Thus, signals representing a"line" and an "edge" are completely separated to be output. A thresholdwidth, where a band having a width less than the threshold width isextracted as a line, can be preset by presetting the gas filter width inthe only-line filter 111. Separation of "gaps" can be made in a similarmanner.

FIG. 60 is a diagram illustrating a concrete construction of anonly-edge extraction filter. In FIG. 60, reference numeral 111 denotesan only-line extraction filter having a construction as indicated inFIG. 58, 112 denotes an edge extraction filter having a construction asindicated in FIG. 52B, and 113 denotes a line eliminate portion whichsubtracts the output of the only-line extraction filter 113 from theoutput of the only-edge extraction filter 112, to output an edge onlysignal. In addition, in the only-edge extraction filter 112, referencenumeral 112a denotes a one-dimensional gr filter.

2.8.7 Multi-Filter

Since, according to the process of "receptive fieldmethod+one-dimensional filter" in the present invention, processing of alarge mask can be performed with high-speed, simultaneous application ofa plurality of filters with different widths (multi-filter) is possible,as explained before. Thus, the "simultaneous extraction of obscureportions and sharp portions contained in an image", which is impossibleby the conventional method, can be performed, and working and movementwith precise recognition of a whole image (screen) is possible. In thevisual cortex of the cerebrum, a plurality of types of filtering withdifferent widths are simultaneously applied, to ensure the extraction ofa line segment.

Hereinafter, embodiments of multi-filters for the three types of linesegments (a line, an edge, and a gap) are explained with theirsimulations.

2.8.7.1 Multi-Filter for Extraction of "Line" and "Gap"

The construction and advantage of the multi-filter are explained for oneof the line extraction filters, which is the most superior multi-filter(FIG. 50B) among the three types of "line" extraction filters indicatedin FIGS. 50A, 50B and 50C. The constructions and advantages of the othermulti-filters will be understood from the explanation for the mostsuperior multi-filter. In addition, "gap" extraction filters can beconstructed in a similar manner by inverting the polarity.

2.8.7.1.1 Prototype Multi-Filter

The prototype of the multi-filter has the construction of FIG. 42, andthe one-dimensional filters 44a˜44d in FIG. 42 are one-dimensional gasfilters of widths W₁ ˜W₄. In the construction, an input image stored inthe input memory 41, is divided into receptive fields by the receptivefield division portion 42, and polar transformation is applied to thereceptive field image by the polar transformation portion 43. Next, theabove one-dimensional gas filters with different filter widths areapplied to the output of the polar transformation in parallel in theρ-direction of the hypercolumn plane. Then, the outputs of theone-dimensional gas filters are synthesized to obtain a multi-filteroutput, and the extraction of a line, an edge, and a gap is performedbased on the multi-filter output. By the above multi-filter,"one-dimensional gas filters with widths W₁ ˜W₄ " are synthesized to"simultaneously extract line segments from a sharp line segment to anobscure line segment". Since the outputs of the filters are simplysummed in the prototype multi-filter, the prototype multi-filter isequivalent to a filter which is made by synthesizing the filters.

2.8.7.1.2 Positive-Negative-Separation-Type Multi-Filter

In the prototype multi-filter, interference occurs when the density oflines becomes great. To avoid interference, it is effective to separatepositive and negative components in each output, and synthesize positivecomponents and negative components, respectively. FIG. 61 is a diagramillustrating a construction of the positive-negative-separation-typemulti-filter wherein positive and negative components in each output areseparated, and positive components and negative components arerespectively synthesized. In the positive-negative-separation-typemulti-filter, a positive component and a negative component in theoutput of each one-dimensional gas filter, which constitutes theprototype multi-filter, are separated, and positive components andnegative components are respectively synthesized to be output. In thepositive-negative-separation-type multi-filter, a positive-negativeseparation circuit 44e˜44h, a positive signal synthesis portion 44i, anda negative signal synthesis portion 44 are provided in the stagefollowing each one-dimensional gas filter 44a˜44d in the prototypemulti-filter.

A simulation result of the positive-negative-separation-typemulti-filter is indicated in FIGS. 62A and 62B, where a line segment SL(FIG. 62A) in the round receptive field CRC, is extracted as a positivemain peak PK accompanied by negative sub-peaks PK1 and PK2 (See FIG.62B).

2.8.7.2 Multi-Filter for "Edge" Extraction

The construction and advantage of the multi-filter are explained for oneof the "edge" extraction filters, which is the most superiormulti-filter (FIG. 50B) among the "edge" extraction filters as indicatedin FIGS. 52A, 52B, 52C, and 52D. The constructions and advantages of theother multi-filters will be understood from the explanation for the mostsuperior multi-filter (FIG. 52B) among the "edge" extraction filters asindicated in FIGS. 52A, 52B, 52C, and 52D. In addition, the other "edge"extraction filters can be constructed in a similar manner.

FIG. 63 is a diagram illustrating the construction of the edgeextraction multi-filter. In FIG. 63, reference numeral 121 denotes areceptive field division portion which divides an input image intoreceptive fields to output the divided images, 122 denotes a polartransformation portion which applies predetermined polar transformationto the receptive field image, 123 denotes a one-dimensional multi-filterportion comprising, one-dimensional gr filters 123a to 123d respectivelyhaving widths W₁₁ to W₁₄, each of the one-dimensional gr filter applyingone-dimensional first differential filter processing to the output ofthe polar transformation, one-dimensional gas filters 123a'˜123d'respectively having widths W₂₁, W₂₂, W₂₃, and W₂₄, independentlyapplying one-dimensional second differential filter processing to theoutput of each one-dimensional gr filter, and a synthesis portion 123efor synthesizing the outputs of the one-dimensional gas filters123a'˜123d' to output the synthesized output.

According to the edge extraction multi-filter, the "simultaneousextraction of obscure portions and sharp portions contained in animage", which is impossible by the conventional method, can beperformed, and working and movement with precise recognition of a wholeimage (screen) is possible. When the width of the gas filter is toosmall, the "line" transformed by the one-dimensional gr filter may bedeemed as a band having a width, so differentiation is performed onedges on both ends of the band, and therefore the output becomes zero atthe main peak. To avoid this problem, the following condition isrequired for each of the plurality of gas filters.

The diameter (W₂₁ ˜W₂₄) of the gas filter≧the width (W₁₁ ˜W₁₄) of the grfilter.

The optimum condition for each of the plurality of gas filters is thediameter (W₂₁ ˜W₂₄) of the gas filter=twice the width of the gr filter.

In addition, to change the construction of the edge extraction filterindicated in FIG. 52A to a multi-filter construction, one-dimensional grfilter processing comprised of a plurality of types of filteringrespectively having different widths (W₁ ˜W₄) must be applied to theoutput of the polar transformation, and the synthesis of the outputs ofthe filtering must be obtained.

2.8.7.3 Positive-Negative-Separation-Type "Edge" Extraction Multi-Filter

When the density of the edges becomes great, interference may occur. Toavoid the interference due to the high-density "edge" output, it iseffective to separate positive and negative components in each output ofthe one-dimensional gr filter, apply one-dimensional gas filterprocessing independently to each of positive and negative components,and synthesize positive and negative components, respectively. FIG. 64is a diagram illustrating the construction of thepositive-negative-separation-type "edge" extraction multi-filter. InFIG. 64, reference numeral 121 denotes a receptive field divisionportion which divides an input image into receptive fields, and outputsthe divided image, 122 denotes a polar transformation portion whichapplies predetermined polar transformation to each receptive fieldimage, 124 denotes a positive-negative-separation-type one-dimensionalmulti-filter portion comprising one-dimensional gr filter 124a-124drespectively having widths W₁₁ ˜W₁₄, and each one-dimensional firstdifferential filter processing to the output of the polartransformation, positive-negative separation circuit 124a'˜124d' whichseparates positive and negative components in the output of eachone-dimensional gr filter output, four pairs of first and secondone-dimensional gas filters 124e, 124e'; 124f, 124f'; 124g, 124g'; 124h,124h', the one-dimensional gas filters in the respective pairs havingwidths W₂₁ ˜W₂₄, and each one-dimensional gas filter independentlyapplying one-dimensional second differential filter processing to thepositive and negative portions in the output of each one-dimensional grfilter, a synthesis portion 124i which synthesizes positive outputs ofthe one-dimensional gas filters, and outputs the result, and a synthesisportion 124j which synthesizes negative outputs of the one-dimensionalgas filters, and outputs the result.

A simulation result of the positive-negative-separation-type "edge"extraction multi-filter is indicated in FIGS. 65A and 65B, where an edgeEDG (FIG. 65A) in the round receptive field CRC, is extracted as a sharppeak PK (FIG. 65B).

2.8.8 Multi-Filter by "Two-Dimensional gas Filter+One-Dimensional grFilter"

In the above, the "one-dimensional gas filter+one-dimensional grfilter"-type multi-filter is explained, wherein the amount of processingis remarkably reduced. Instead, the combination of a "two-dimensionalgas filter+a one-dimensional gr filter" can be used. The function andadvantage of the latter filter is the same as the former filter, and theconstruction of the latter filter is indicated in FIGS. 66A and 66B.

2.8.8.1 Line Extraction Multi-Filter

FIG. 66A is a diagram illustrating the construction of the "line"extraction multi-filter, which comprises a plurality of two-dimensionalgas filters (W₁ ˜W₄), respectively applying two-dimensional seconddifferential filter processing to an image, and the outputs of therespective filters are synthesized. The synthesized image is dividedinto receptive fields by the receptive field division portion, and polartransformation is applied to each receptive field image by the polartransformation portion to extract a line.

2.8.8.2 Gap Extraction Multi-filter

The gap extraction multi-filter is obtained by inverting the polarity in2.8.8.1.

2.8.8.3 Edge Extraction Multi-Filter

FIG. 66B is a diagram illustrating the construction of the "edge"extraction multi-filter, which comprises a plurality of two-dimensionalgas filters, respectively applying two-dimensional second differentialfilter processing to an input image. In the construction of FIG. 66B,further, the output of each two-dimensional gas filter is independentlydivided into receptive fields, polar transformation is applied to eachreceptive field image, one-dimensional gr filter processing is appliedto each output of the polar transformation, and the outputs of theone-dimensional gr filters are synthesized to output an edge signal.

According to the above edge extraction multi-filter, although the outputequivalent to the output of the "one-dimensional gasfilter+one-dimensional gr filter"-type multi-filter is obtained, theamount of processing in the two-dimensional gas filter increases with asquare of the filter size.

2.8.9 Example Variation of Edge Multi-Filter

As the edge multi-filters, the edge multi-filters indicated in FIGS. 63and 64 are most superior, the multi-filter indicated in FIG. 67A whereinthe gas filter is fixed, is the second superior. In the visualinformation processing of mammalian, this multi-filter is adopted sincethe output (the retina and the lateral geniculate nucleus) of thetwo-dimensional gas filter (diameter W₁₀) can serve for common use. Inaddition, for the above reason, it is necessary that the followingcondition exists:

the diameter (W₁₀) of two-dimensional gas filter≧max (the width ofone-dimensional gr filter).

The amount of processing can be further reduced in the construction ofFIG. 67B, wherein the above two-dimensional gas filter is modified to aone-dimensional form.

2.9 Extension of One-Dimensional Filter

Although the above explanations are made for the case of "polartransformation on a sphere" and "the mapping method as a portionthereof" for ease in understanding, the process and construction of"receptive field division→polar transformation→one-dimensional filter"according to the present invention, can operate for general polartransformation, for example, polar transformation on a cylinder, polartransformation on a plane, and the like, in the same way, and thereforehigh-speed filtering, which is difficult by the two-dimensional filter,and filtering of an arbitrary function form, which is impossible in thetwo-dimensional filter, are made possible. This is because, according tothe receptive field method wherein an image is divided into small areas,a projection axis ψ in a broad sense and an axis ρ perpendicular to theaxis ψ can be locally defined for general polar transformation, andtherefore a one-dimensional filter can be applied to obtain theadvantage as explained above.

Although application of a one-dimensional filter in the direction of the"location axis ρ of a line segment" is explained above, furtherfiltering in the "angle orientation θ of a line segment" is alsopossible. Due to this filtering, extraction of a circle which has acommon center with the receptive field, is possible. The reason isbecause, a trace generated by varying θ only on a dual plane,corresponds to an "envelope curve of a circle" in the receptive field.The above circle can be extracted by extracting a group of peaks in arow in the θ-direction. In this special case, a group of radial linespassing through the center of the receptive field at ρ=0.

Although, in the above explanations, an image input through a camera anda lens is divided into small areas, polar transformation is applied tothe image in each small area, one-dimensional filter processing isapplied to the output of the polar transformation, and image processingis performed based on the result of the one-dimensional filterprocessing, the present invention can be applied to an arbitrary imageother than the images input through a camera and a lens. For example,the hypercolumn image may be divided into small areas. Then, polartransformation is applied to the image in each small area,one-dimensional filter processing is applied to the output of the polartransformation, and image processing is performed based on the result ofthe one-dimensional filter processing.

Although the present invention is explained with the embodiments above,various types of variations are possible within the spirits of thepresent invention, as stated in the claims, and the scope of the presentinvention includes these types of variations.

2.10 Advantage of Second Aspect of Present Invention

2.10.1 Since the input image of a size equal to N×N pixels is dividedinto receptive field images of a size equal to m×m pixels, polartransformation is applied to the image in each small area, predeterminedone-dimensional filter processing is applied to the output of the polartransformation to extract a characteristic feature, and only applicationof one-dimensional filter is required, so the amount of processing isremarkably reduced by about (2/filter diameter pixels), compared withthe amount of processing in the application of the conventionaltwo-dimensional filter, and thus high-speed processing becomes possible.Filtering of a large size whereby processing of an obscure image andfine filtering of line segments can be performed with the same amount ofprocessing as the amount of conventional processing, can be realized,and line segments can be precisely extracted from the obscure image.

2.10.2 Since a one-dimensional filter is applied after the projection inall directions (polar transformation) is performed, and isotropicprocessing is completed by the projection, the one-dimensional filterapplied after the projection may be the same for all the orientations,and therefore an arbitrary filter can be applied. Therefore, theone-dimensional filter can be the one-dimensional odd-function filterwhich is indispensable for the edge extraction, and thus extraction ofedges becomes possible while it is impossible by the conventionalfilter.

2.10.3 The one-dimensional filter can be constructed by skeletonfilters, and efficient differentiation processing can be performed bythe skeleton filters. In addition to the above factor contributing tospeed-up, processing speed can be made faster by a factor correspondingto "the number of pixels in the filter width". Namely, extraction of thecharacteristic feature, for example, lines, edges, gaps, and the like,can be performed with high-speed.

2.10.4 Since a plurality of types of one-dimensional filtersrespectively having different widths are simultaneously applied to theoutput of the polar transformation, and the outputs of the plurality oftypes of one-dimensional filters are synthesized (multi-filter), thesimultaneous extraction of obscure portions and sharp portions containedin an image, and working and movement with precise recognition of awhole image (screen) is possible.

2.10.5 Since an object image is divided into receptive field images,polar transformation is applied to each receptive field image,one-dimensional gas (Gaussian) filter processing is applied to theoutput of the polar transformation, so extraction of a line can beperformed based on the result of the one-dimensional gas processing, andextraction of a gap can be performed by inverting the polarity of theresult of the one-dimensional gas processing. Further, since extractionof lines and gaps is performed by selecting a portion in the vicinity ofa peak in the output of the one-dimensional Gaussian filter which isapplied after the polar transformation, a sharp line segment and gap canbe extracted.

2.10.6 Since an object image is divided into receptive field images,polar transformation is applied to each receptive field image,one-dimensional gr (gradient) filter processing is applied to the outputof the polar transformation, or one-dimensional gr (gradient) filterprocessing and one-dimensional gas filter processing are applied to theoutput of the polar transformation (one-dimensional multi-stage filterprocessing), so extraction of an edge can be performed, while it isconventionally impossible. Further, since a plurality of types ofone-dimensional gr filters respectively having different widths aresimultaneously applied to the output of the polar transformation, or aplurality of types of one-dimensional gr filters respectively havingdifferent widths, and a plurality of types of one-dimensional gasfilters respectively having different widths are applied, respectivelyand in turn, and then the outputs of the plurality of filters aresynthesized to be output, simultaneous extraction of edges in obscureportions and sharp portions contained in an image, is possible.

2.10.7 Since positive and negative components in the output in eachstage are separated with one-dimensional multi-stage filtering, finecharacteristic features such as a narrow band and high density of linesand edges can be extracted while suppressing interference.

3. Third Aspect of The Present Invention

3.1 Basic Construction of Third Aspect of Present Invention

FIG. 68 is a diagram illustrating the basic construction of the thirdaspect of the present invention (receptive field method).

In FIG. 68, reference numerals 15' and 25' each denote a polartransformation portion which applies polar transformation processing tofirst and second image data, 17' and 27' each denote a one-dimensionalfilter which applies one-dimensional filter processing to a result ofthe polar transformation, 18' and 28' each denote a ρθ dual plane(hypercolumn memory) which stores a result of the filter processing(hypercolumn image), and 30 denotes a correlation processing portionwhich performs correlation processing between data mapped onto therespective dual planes.

The polar transformation portions 15' and 25' apply polar transformationprocessing to the first and second image data, and thus projection ontodual planes 18' and 28' is performed. The one-dimensional filters 17'and 27' apply one-dimensional filters 17' and 27' to the result of thepolar transformation processing result, and thus mapping onto dualplanes 18' and 28' is performed. The correlation processing portion 30obtains a correlation amount (correlation parameter) by using as elementparameters a location (ρ,θ) of the mapped data on the dual plane, and ashift amount between the mapped data to which correlation processing isto be performed. The correlation processing portion 30 further obtains apoint at which a characteristic correlation parameter (for example, alocal maximum value) is located, and measures a variable (a binocularparallax, a movement direction, a velocity, and the like) whichdetermines relationship between characteristic features of therespective images based on values of the element parameters whichprovide the characteristic correlation parameter. Since a tangentialline of a contour included in the first and second images (thetangential line is a straight line when the contour is a straight line)is transformed to a point decreasing the dimension by polartransformation, the two-dimensional problem of the tangential line istransformed to a one-dimensional problem, and thus, determinationprocessing of corresponding portions in a plurality of figures bycorrelation processing can be made simple, and further the determinationprocessing can be performed precisely with a smaller amount ofprocessing to realize the function of the binocular stereopsis.

In addition, the amount of processing can be remarkably reduced byperforming polar transformation processing, filter processing, andcorrelation processing on the receptive field images generated bydividing an image (screen) into receptive fields which are small areas.

Further, when the respective receptive field images belong to differentimages (screens) captured by two or three cameras, the function of thebinocular stereopsis and the three-dimensional view by three eyes can berealized. When the respective receptive field images belong to differentimages captured at different images, the movement direction and movementvelocity of a characteristic feature (a line, a corner, and the like) ineach receptive field can be measured, and therefore it is possible tomove while capturing an object at the center of the field of view, andthis technique can be applied to a moving robot and an unmanned vehicle.

When the receptive field images are images in different receptive fieldsin the same image (screen), or when the receptive field images areimages in the same receptive field image, texture analysis can beperformed wherein, for example, a degree of repetition of the samedesign pattern in an image can be examined.

Further, the binocular stereopsis, the pursuit of a movement object, andthe texture analysis can be performed more surely by performingcorrelation processing between a plurality of receptive field images foreach color, for each color difference signal, or for each primary color.

When one-dimensional Gaussian filter processing is performed after thepolar transformation, or two-dimensional Gaussian filter processing isperformed before the polar transformation processing, correspondinglines or gaps in a plurality of figures can be obtained. Further,one-dimensional gradient filter processing and one-dimensional Gaussianfilter processing are performed after the polar transformationprocessing; or two-dimensional Gaussian filter processing is performedbefore the polar transformation processing, and one-dimensional gradientfilter processing is performed after polar transformation processing,corresponding edges in a plurality of figures can be obtained. Then, thelocation, an orientation, a binocular parallax, a movement direction,and a movement velocity of these figure elements can be obtained.

When a plurality of receptive field images belong to spatially differentimages, it is possible to extract a tangential line moving with changingits orientation by calculating correlation parameter C(ρ,θ,σ) in theθ-direction, or two-dimensional correlation parameter C(ρ,θ,σ₁,σ₂) onthe (ρ,θ) plane.

When a plurality of receptive field images belongs to different image(screen) at different times, a location, an orientation, and a velocityof a translating tangential line can be quantitatively obtained bycalculating correlation parameter C(ρ,θ,τ) in the ρ-direction. Inaddition, a location, an orientation, a velocity, and the like of atangential line passing through the center of the receptive field, bycalculating correlation parameter C(ρ,θ,τ) in the θ-direction. Further,it is possible to quantitatively measure a location, an orientation, avelocity, a rotation velocity, and the like of a line moving withchanging its orientation by calculating two-dimensional correlationparameter C(ρ,θ,τ₁,τ₂) in the (ρ,θ) plane.

Further, the capacity of memory storing correlation parameters can bereduced by an amount corresponding to one or two axes: by projectingcorrelation parameter C(ρ,θ,σ) in a σ-axis direction, where σ denotes aspatial shift amount; projecting the correlation parameter C(ρ,θ,σ) in aρ-direction, where σ denotes a tangential line location; projectingcorrelation parameter C(ρ,θ,σ) in a θ-direction, where θ denotes atangential line orientation; or projecting correlation parameterC(ρ,θ,σ) in the orientations of arbitrary two of the above axes. Inaddition, a desired value among a location, a binocular parallax, and anorientation of a tangential line, can be obtained by selecting theprojection direction.

Further, the capacity of memory storing correlation parameters can bereduced by an amount corresponding to one or two axes: by projectingcorrelation parameter C(ρ,θ,τ) in a τ-axis direction, where τ denotes atime shift amount; projecting the correlation parameter C(ρ,θ,τ) in aρ-direction, where ρ denotes a tangential line location; projectingcorrelation parameter C(ρ,θ,τ) in a θ-direction, where θ denotes atangential line orientation; or projecting correlation parameterC(ρ,θ,τ) in the directions of arbitrary two of the above axes. Inaddition, a desired value among a location, an orientation, a velocityof translation, and a rotation velocity of a tangential line, can beobtained by selecting the projection direction.

Further, by performing polar transformation processing on the receptivefield image to map onto a ρ-θ dual plane, picking up a combination ofmapped data a(ρ,θ) on the dual plane, and mapped data b(ρ,θ) on anotherdual plane, where the coordinate values are shifted in the ρ-directionby a predetermined amount, and calculating a sum of products of thecombination data, precise filtering can be performed, and preferably,this technique is applicable to "extraction of a characteristic featurewhich can be seen by right and left eyes in the same way" and "pursuitof the same characteristic feature in the current and preceding images".

From mapped data a(ρ,θ) on a dual plane, mapped data b(ρ,θ) after themapped data (ρ,θ) is shifted in the ρ-direction or the θ-direction, issubtracted. Then, the subtraction is repeated varying the shift amountto obtain the subtraction result as correlation parameters. Thus, thebinocular stereopsis and trace of an object the contour of which isobscure, based on slow variation of brightness and hue, and the like.

3.2 Principle of Binocular Stereopsis

In the conventional binocular stereopsis, corresponding "figures" inimages are searched. It is difficult to stably determine thecorrespondence by the conventional computers which are poor at theprocessing of figures. Therefore, determination of the correspondence isperformed based on the simplest characteristic feature, i.e., "atangential line" of a figure (an element of a contour when cutting thecontour into very small lengths, and the element is a straight line whenthe contour is a straight line). According to this "tangential linemethod" comparison of tangential lines (straight line) projected ontotwo eyes should be performed. The procedure is simple, and thus stabledetermination of the correspondence can be performed. In the explanationhereinafter, when terms such as a tangential line, a contour tangentialline, a contour line have the same meaning as long as consideration ismade within a receptive field image in each receptive field as a smallarea.

Although the treated figure becomes simple due to the above comparisonof tangential lines, it is necessary to determine the correspondence ofa tangential line, and it takes time for the processing. When performingpolar transformation on an input image, and mapping thepolar-transformed image onto a dual plane which has an orientation θ anda location ρ as coordinate-axes, "tangential lines" in the right andleft images are transformed to "a point" with reducing the dimension.Therefore, the two-dimensional problem of the correspondence oftangential lines, can be processed as a one-dimensional problem toremarkably reduce the amount of processing.

FIG. 69 is a diagram for explaining the principle of the binocularstereopsis according to the present invention.

(1) An image SL_(L) of a "tangential line" in a space in the input imageIM_(L) seen by the left eye is offset in parallel from an image SL_(R)of the "tangential line" in the input image IM_(R) seen by the righteye.

(2) On the other hand, in the ρ-θ dual planes HCP_(L) and HCP_(R) ontowhich a hypercolumn image is mapped, where the hypercolumn image isobtained by polar-transforming an input image, a group of parallel linesinclined by an angle θ_(P) is transformed to a series of points on theρ-axis, where θ=θ_(P).

(3) Therefore, the "tangential lines" SL_(L) and SL_(R) seen with theoffset in parallel in the input images IM_(L) and IM_(R) by the rightand left eyes, are transformed to the points P_(L) and P_(R) on the dualplanes HCP_(L) and HCP_(R) corresponding to the same θ_(P) on theρ-axis.

(4) By obtaining a distance σ_(P) between the two points, an amount oftranslation of the "tangential lines" SL_(L) and SL_(R) in the right andleft images IM_(L) and IM_(R), i.e., a binocular parallax is determined.

(5) Based on the above distance σ_(P) and the space between two eyes,the spatial depth of the "tangential lines" can be determined, and thefunction of the binocular stereopsis is realized.

Thus, by performing polar transformation on the input image, comparisonof the "tangential lines" on the "two-dimensional space" can besimplified to the one-dimensional processing for comparing the "point"on the ρ-axis, the determination of the correspondence can be performedwith small amount of processing.

3.3 Basic Principle of Third Aspect of Present Invention Summarizing theabove explanations, the basic construction of the binocular stereopsisis expressed as below.

Input image→polar transformation→one-dimensional correlation processing

As explained above, the polar transformation is an operation oftransforming a "tangential line" to a "point" to perform the comparisonin a one-dimensional space. The one-dimensional correlation processingis an operation of extracting corresponding "points" from "a series ofpoints" corresponding to the same θ on the ρ-axis. When mapped data onpolar-transformed dual planes of two eyes are respectively denoted byL(ρ,θ) and R(ρ,θ), and a spacing between mapped data (shift amount) isdenoted by σ, the correlation amount (correlation parameter) iscalculated by the following equation

    C(ρ,θ,σ)=L(ρ,θ)·R(ρ+σ,θ).(101)

By obtaining values (ρ_(P),θ_(P),σ_(P)) of the element parameters ρ,θ,σat which the correlation amount C(ρ,θ,σ) becomes locally maximum,corresponding tangential lines can be determined based on the values(ρ_(P),θ_(P),σ_(P)) of the element parameters, and a location, anorientation, and a binocular parallax of the tangential lines arequantitatively determined. In addition to the asymmetric-typecorrelation calculation by the equation (101), the correlation parametercan be obtained by symmetric-type correlation calculation as indicatedby the equation (101)',

    C(ρ,θ,σ)=L(ρ-σ,θ)·R(ρ+σ,θ)                                                   (101)'

Hereinafter, basically, explanations are given based on the assumptionthat the correlation parameter is calculated by the asymmetric-typecorrelation calculation according to the equation (101). In addition,terms such as correlation amount, correlation parameter, and correlationresult, are used with the same meaning.

3.4 Embodiment of Correlation Processing Apparatus

3.4.1 Whole Construction

FIG. 70 is a diagram illustrating the construction of the embodiment ofthe third aspect of the present invention, wherein a location, anorientation, and a parallax of the corresponding tangential linescontained in two images are obtained by the procedure of: inputimage→receptive field division→polar transformation one-dimensionalcorrelation processing. In FIG. 70, reference numerals 312 and 322 eachdenote an input memory for storing input images IM_(L) and IM_(R) seenby right and left eyes, 313 and 323 each denote a receptive-fieldcut-out circuit which cuts out and outputs an image (receptive fieldimage) in each receptive field in turn when an input plane is dividedinto receptive fields which are small areas of a size m×m pixels, 314and 324 each denote a receptive field memory for storing an image(receptive field image) in each receptive field, 315 and 325 each denotea polar transformation circuit for applying polar transformation to eachreceptive field image, 316 and 326 each denote a polar transformationhypercolumn memory for storing images (hypercolumn image) which is polartransformed onto dual planes (hypercolumn plane). The polartransformation hypercolumn memory 316 and 326 comprises storage areas(hypercolumn cells) containing ρ_(max) areas in the ρ-direction andθ_(max) areas in the θ-direction, and ρ_(max) ×θ_(max) areas as a total.

In addition, reference numerals 317 and 327 each denote aone-dimensional filter circuit for applying one-dimensional filterprocessing to the hypercolumn images obtained by the polartransformation, 318 and 328 each denote a dual plane (hypercolumnmemory) for storing hypercolumn images to which filter processing isapplied, 330 denotes a correlation processing portion. In thecorrelation processing portion 330, reference numeral 331 denotes acorrelation calculation portion for performing one-dimensionalcorrelation calculation according to the equation (101), 332 denotes acorrelation parameter storage portion for storing correlation values(correlation parameters) having a dimension of ρ-σ-θ, 333 denotes a peakdetect portion for scanning the data stored in the correlation parameterstorage portion to detect a local maximum point (ρ_(P),θ_(P),σ_(P)) .

The polar transformation circuit 315, 325 applies polar transformationto each pixel in the receptive field, i.e., transforms each pixel to acorresponding great circle to store the transformed data in the polartransformation hypercolumn memory 316, 326. In practice, since the sizeof the receptive field is small, the polar transformation isapproximated as a transformation of "pixels→sine wave", and each pixelin the receptive field is transformed to a corresponding sine curve on ahypercolumn to store the transformed curve in the polar transformationhypercolumn memory 316, 326.

The one-dimensional filter circuit 317, 327 is provided for extracting aline, an edge, and a gap with a small amount of processing. Foremphasizing contours and extracting characteristic features, usually,two-dimensional convolution filter is applied to the image data, andthen polar transformation is performed. However, according to thisconvolution method, an amount of processing is equal to a² is necessaryfor each input point when the filter size is equal to a, and thereforethe amount of processing increases with the increase in the filter size.However, since the process of "two-dimensional convolution filter+polartransformation" is equivalent to the process of "one-dimensional filterprocessing in the ρ-direction after polar transformation",one-dimensional filter is applied after polar transformation in theembodiment of FIG. 70. According to this process, the amount ofprocessing becomes small as a due to one-dimensional filtering, and theamount of processing is reduced by about 2/a compared with theconvolution method.

3.4.2 Correlation Calculation

The correlation calculation portion 331 multiplies the mapped dataL(ρ,θ) on the dual plane 318 by the left eye, by the mapped dataR(ρ+σ,θ) on the dual plane 328 by the right eye, which is shifted by ain the ρ-direction from the mapped data L(ρ,θ), and stores themultiplication result in the correlation parameter storage portion 332.Then, the multiplication and storage are repeated with varying the shiftamount σ as 0˜σ_(max), ρ as 0˜ρ_(max) (for a full width of thehypercolumn in the ρ-direction), θ as 0˜θ_(max) (for a full width of thehypercolumn in the θ-direction).

FIG. 71 is a flowchart of the correlation calculation processing. InFIG. 71, for the correlation calculation, initially, parameters are setas 0→θ, 0→ρ, 0→σ (steps 1101˜1103). Next, the correlation parameter iscalculated according to the equation (101), and the calculatedcorrelation parameter is stored in the correlation parameter storageportion 332 (steps 1104, 1105). Then, σ is incremented, and it isdetermined whether or not σ>σ_(max) (steps 1106, 1107). When it isdetermined that σ≦σ_(max), the operation goes back to step 1104 torepeat the above processing. When it is determined that σ>σ_(max), ρ isincremented, and it is determined whether or not ρ>ρ_(max) (steps 1108,1109). When it is determined that ρ≦ρ_(max), the operation goes back tostep 1103 to repeat the above processing. When it is determined thatρ>ρ_(max), θ is incremented, and it is determined whether or notθ>θ_(max) (steps 1110, 1111). When it is determined that θ≦θ_(max), theoperation goes back to step 1102 to repeat the above processing. When itis determined that θ>θ_(max), the correlation calculation processing iscompleted.

The polar transformation circuit 315 (as well as the polartransformation circuit 325) is the same as the polar transformationcircuit indicated in FIG. 33.

The one-dimensional filter circuit 317 (as well as the one-dimensionalfilter circuit 327), similar to the one-dimensional filter circuitindicated in FIG. 34, comprises: a reading control portion 17a forreading amplitudes stored in the polar transformation hypercolumn memory316 from ρ_(max) addresses for a constant θ (for example, see thehatched portion), and outputting the same; a one-dimensional filterportion 17b for applying one-dimensional filtering processing to eachamplitude read as above; and a writing control portion 17c for writing aresult of the one-dimensional filtering processing in a storage area inthe dual plane (hypercolumn memory) 18. The one-dimensional filterportion 17b has, for example, a FIR-type digital filter construction,and comprises a one-dimensional filter memory 17b-1 and asum-of-products circuit 17b-2.

3.4.3 Simulation Result

FIGS. 72 to 74 are diagrams for explaining a simulation result wherein aone-dimensional Gaussian filter is used as the one-dimensional filter317, 327 in FIG. 70 for extracting a line. By the polar transformationand the one-dimensional Gaussian filter processing, the "line" in theimages IM_(L) and IM_(R), is transformed to a "point" in the dual planewhich has a location ρ and an orientation θ of the line ascoordinate-axes line.

In FIGS. 72A and 72B, receptive fields seen by the right and left eyesare indicated in the circles C_(L) and C_(R), respectively. The image ineach circle is an image projected on an eye, and a figure comprised ofcrossed lines, which is encircled in a rectangular in a scene of anindustrial plant, is shot, where intensity of signals of all data isindicted by contour lines.

FIGS. 73A and 73B are diagrams illustrating a hypercolumn image on theρ-θ dual plane, where the hypercolumn image is obtained by theabove-mentioned processing of "polar transformation+one-dimensionalGaussian filter", and two lines constituting the crossed lines aretransformed to two points P_(L1), P_(L2) ; P_(R1), P_(R2) respectivelylocated on the ρ-axes at 90° and 180°. These peaks are sharpened by theone-dimensional filtering, and as the secondary effect thereof, areaccompanied by negative (opposite to the polarity of the main peak)sub-peaks P_(L1) ', P_(L2) '; P_(R1) ', P_(R2) ' on both sides. Thesepeaks are effective to sharpen a processing result of the nextcorrelation.

FIG. 74 is a diagram illustrating a result of the correlation amount(correlation parameter) C(ρ,θ,σ) obtained from dual plane data of theright and left eyes according to the equation (101), where the result isdeveloped on the ρ-σ-θ plane, the ρ-σ-θ plane which has the horizontalaxis ρ (location), the vertical axis σ (binocular parallax), and thedepth θ (orientation), the correlation values are indicated by contourlines, and the ρ-σ plane of θ=90° is indicated on the top of the ρ-σ-θplane for convenience of explanation.

By obtaining ρ_(P),θ_(P),σ_(P) where the correlation parameter C(ρ,θ,σ)has a peak (local maximum), correspondence relationship between imagesof the crossed lines projected onto the right and left eyes isquantitatively obtained, where ρ_(P),θ_(P),σ_(P) indicate a location andorientations of lines, and binocular parallaxes, respectively. Theparallax obtained from images of the vertical line (θ=90°) of thecrossed lines seen by the right and left eyes, is 1°, and the distanceto the vertical line is calculated as 360 cm when the space between twoeyes is equal to 6 cm. Thus, completion of precise binocular stereopsisis confirmed.

3.4.4 Evaluation

As explained above, according to the basic processing (polartransformation+one-dimensional filtering+one-dimensional correlationprocessing), the following advantages are obtained.

3.4.4.1 Advantage of Comparison of Characteristic Features of Figure byTangential Lines

It is difficult to find the same figure from the right and left imagesbecause the current image processing technique is poor at the processingof "a figure". The processing of obtaining correspondence of figuresbecomes simplest when the correspondence is obtained for tangentiallines constituting contours of the figures, because correspondence ofthe figures is obtained as correspondence of straight lines.

3.4.4.2 Advantage of One-Dimensional Processing for Two-DimensionalCorrespondence of Tangential Lines Conventionally, two-dimensionalprocessing is necessary to determine correspondence between tangentiallines in the right and left images. However, by performing polartransformation processing to map onto the dual plane, before the processfor correspondence, a group of parallel lines are transformed to aseries of points on the ρ-axis. Due to the characteristic of the polartransformation, the correspondence between tangential lines istransformed to processing for one-dimensional correlation on the ρ-axis,and therefore the amount of processing is remarkably reduced, and thestability of the obtained correspondence is improved

3.4.4.3 Advantage of High-Degree Filtering

The high-degree one-dimensional filtering processing, disclosed in thepatent, can be performed after polar transformation. As the result ofthis processing, sure determination of the correspondence by theprocedure of: polar transformation→one-dimensionalfilter→one-dimensional correlation processing, is possible, and thusrespective types of correlation parameters can be measured finely. Thetypes and functions of the one-dimensional filtering are as follows.

•Determination of correspondence between edges by odd-functionfiltering: The edge is the most important characteristic feature of animage. The extraction of an edge (a border of brightness) is possible bythe above procedure while the extraction of an edge is difficult by theconventional two-dimensional convolution filter. This is becauseapplication of the odd-function filter is possible due to theapplication of the polar transformation before the odd-functionone-dimensional filtering.

•Stable Determination of Correspondence by Multi-Filtering: A pluralityof filters with different widths can be applied simultaneously topolar-transformed data (multi-filter). Thereby, the simultaneousextraction of "obscure characteristic features" and "fine characteristicfeatures" in the image, and stable determination of correspondencebecomes possible, while these are difficult by the conventional method.

•High-Speed Processing by Skeleton-Type One-Dimensional Filtering: Topolar-transformed data, the most efficient skeleton-type one-dimensionalfilter can be applied. By use of the skeleton-type one-dimensionalfilter, the above odd-function filter and the multi-filter can berealized with high-speed.

3.5 Extension of Third Aspect of Present Invention

In the above explanations, the principle and the advantages of thecorrelation filtering are explained for the binocular stereopsis as anexample. However, the essence of the third aspect of the presentinvention is to determine a degree of correlation between images, andthere are many applications of the third aspect of the presentinvention, in addition to the correlation between two eyes, includingpursuit of a moving object by correlation between images at differenttimes, texture analysis wherein a degree of appearance of the samedesign pattern in the same image, and the like. Hereinafter, theextension of the correlation filtering according to the presentinvention is explained.

3.5.1 Basic Construction

FIG. 75 is a basic block diagram of the correlation filtering, whereinpolar transformation processing is applied to input data A and B (351a,351b), one-dimensional filtering processing is applied to the result ofthe polar transformation to map the same onto correlation dual planes(351a, 352b), and correlation processing is performed between the mappeddata on the respective dual planes (3530).

It is desirable to divide the input data into receptive fields, but thereceptive field division is not necessary. The one-dimensional filterprocessing can be applied as needed, where the types of the filtersexplained before can be used. In addition, the input data is not limitedto the image from the camera, for example, and the input data may bedata which is generated by polar transformation is applied to the imagefrom the camera. Further, the input data is not limited to two images,and may be three or more images.

Hereinafter, it is assumed that the receptive field division, and theone-dimensional filter processing, are applied.

3.5.2 Images to be Correlated

As the images which are to be correlated, i.e., as the input data A andB from which the above correlation is obtained in the basicconstruction, the following types are considered, and the correlationparameters respectively specific to the types of the input data A and B,can be measured.

3.5.2.1 Correlation between Different Images

3.5.2.1.1 Correlation between Spatially Different Images

In the case where input data A and B are spatially different images, forexample, images taken by different cameras, and correlation filtering isperformed on the input data, correlation parameter of a characteristicfeature between spatially different images can be extracted. Inparticular, the binocular stereopsis and the three-eye three-dimensionalview are useful examples.

FIGS. 76A and 76B are block diagrams of the correlation filteringbetween spatially different images. FIG. 76A indicates the case whereinimages are input from two cameras, and FIG. 76B indicates the casewherein images are input from three cameras.

•In the case of images input from two cameras (FIG. 76A)

In the case of two cameras, image data IM_(L) and IM_(R) of a frametaken by a left camera CML and a right camera CMR are respectivelydivided into receptive field images (350a, 350b), polar transformationprocessing is applied to each receptive field image IM_(L) ' and IM_(R)' (351a, 351b), one-dimensional filtering processing is applied to theresult of the polar transformation to map onto dual planes (352a, 352b),and correlation processing is performed between data mapped onto therespective dual planes (3531). The aforementioned binocular stereopsiscorresponds to this operation.

•In the case of images input from three cameras (FIG. 76B)

Since there are two angles when seeing an object by the binocularstereopsis, the correspondence may be erroneously determined. From thepoint of view of engineering, use of three eyes is also possible. Whenseeing an object from three angles by three cameras, the possibility ofan optical illusion is remarkably reduced. When arranging the threecameras at the apexes of a triangle, the optical illusion can be reducedin all directions. In the case of three cameras, image data IM_(L),IM_(C), and IM_(R) of one frame taken by a left camera CML, a centercamera CMC, and a right camera CMR, are respectively divided intoreceptive field images (350a˜350c), polar transformation processing isapplied to each receptive field image IM_(L) ', IM_(C) ', and IM_(R) '(351a˜351c), one-dimensional filtering processing is applied to theresult of the polar transformation to map onto dual planes (351a˜352b),and correlation processing is performed between data mapped onto therespective dual planes (3532). In this case, when data to which thepolar transformation and the one-dimensional filtering processing areapplied, corresponding to the left camera CML, the center camera CMC,and the right camera CMR, are denoted by L(ρ,θ), C(ρ,θ), and R(ρ,θ),respectively, the correlation amount is given by, for example,L(ρ+σ_(L),θ) C(ρ,θ) R(ρ+θ_(R),θ), where σ_(L) and σ_(R) each denote ashift amount.

3.5.2.1.2 Correlation between Different Images Regarding Time

The case wherein the input data A and B in the above basic constructionare images shot at different times, for example, images taken by amoving camera, and correlation filtering is applied to the input data,is explained below.

Since, according to the above correlation filtering, for example, amovement direction and a movement velocity of a characteristic feature(a corner and a contour line of an object, and the like) in a receptivefield can be measured, it is possible to move while capturing the objectin the center of the field of view, and this technique can be applied toa moving robot and an unmanned vehicle.

In the first visual field in the cerebrum, cells for detecting amovement direction of a tangential line in a receptive field exist, asignal is sent to a control portion (the superior colliculus) forcontrolling eyeball movement to capture an object in the center of thefield of view, and another signal is sent to a higher order visual fieldto perform three-dimensional measurement of the movement of a sceneprojected on the field of view.

FIG. 77 is a block diagram illustrating a construction for thecorrelation filtering between images at different times. Image dataIM_(L) of one frame taken by a left camera CML is not delayed, whileimage data IM_(R) of one frame taken by a right camera CMR is delayed inthe delay portion (349), each of the image data IM_(L) without delay,and the delayed image data IM_(R) ', is divided into receptive fields(350a, 350a'), polar transformation is applied to each receptive fieldimage IM_(L), IM_(R) ' (351a, 351a'), one-dimensional filteringprocessing is applied to the result of the polar transformation to maponto dual planes (352a, 352a'), and correlation processing is performedbetween data mapped onto the dual planes (3534). Namely, the input imagedata is divided into delayed image data and non-delayed image data, andthese image data are input into the basic construction of FIG. 75, andthe correlation parameter regarding time is obtained. The operation ofthe construction of FIG. 77 operates in the same way as theaforementioned binocular stereopsis except that the delay portion isprovided in the construction of FIG. 77.

3.5.2.1.3 Correlation between Different Images Regarding Space and Time

The case wherein correlation filtering is performed on images taken by aplurality of cameras while moving respective cameras, is explainedbelow. When the number of the cameras is equal to two, this correlationfiltering is equivalent to correlation filtering combining the binocularstereopsis space correlation filter and the time correlation filter.

FIG. 78 is a block diagram illustrating a construction for correlationfiltering, between images which are different regarding time and space.Image data IM_(L) of one frame taken by a left camera CML is delayed inthe delay portion (349a), each of the image data IM_(L) without delay,and the delayed image data IM_(L) ', is divided into receptive fields(350a, 350a'), polar transformation is applied to each receptive fieldimage IM_(L), IM_(L) ' (351a, 351a'), and one-dimensional filteringprocessing is applied to the result of the polar transformation to maponto dual planes (352a, 352a'). Parallel to the above, image data IM_(R)of one frame taken by a right camera CMR is delayed in the delay portion(349b), each of the image data IM_(R) without delay, and the delayedimage data IM_(R) ', is divided into receptive fields (350b, 350b'),polar transformation is applied to each receptive field image IM_(R),IM_(R) ' (351b, 351b'), and one-dimensional filtering processing isapplied to the result of the polar transformation to map onto dualplanes (352b, 35b'). Then, correlation processing is performed betweendata mapped onto the respective dual planes (3535).

When an object is pursued by extracting a parallax by the binocularstereopsis according to the correlation filtering between imagesdifferent in time and space, complex functions as realized in the humanbody become possible. Although the existence of the filtering functionwherein the time and space are fused, is known in psychology as"Spatiotemporal filter", according to the technique explained in thisitem, the above function is realized in the field of engineering.

Although correlation filtering between different images is explainedabove, correlation within the same image is also effective.

3.5.2.2 Correlation between Same Images

3.5.2.2.1 Correlation between Receptive Fields

FIG. 79 is a block diagram illustrating a construction for correlationbetween respective receptive fields within the same image. An image ofone frame taken by camera CM is stored in the input memory IMM, is cutout into receptive field images IM_(A), IM_(B), IM_(N) corresponding toreceptive fields A, B, . . . , N by the control portion (348), polartransformation is applied to each receptive field image IM_(A), IM_(B),. . . IM_(N) (351a˜351n), one-dimensional filtering processing isapplied to the result of the polar transformation to map onto dualplanes (352a˜352n), and correlation processing is performed between datamapped onto the respective dual planes (3536).

In the case of the correlation between respective receptive fieldswithin the same image, a characteristic feature in a receptive fieldimage IM_(A) can be compared precisely with characteristic features inreceptive fields images IM_(B), . . . IM_(N), and texture analysisbecomes possible. The processing of the above correlation is the same asthe binocular stereopsis, wherein right and left eyes are replaced withthe receptive fields, and is performed in the same way as theaforementioned simulation.

3.5.2.2.2 Correlation within Receptive Field

Although correlation between receptive fields is explained as above,correlation within the same receptive field in the same image ispossible. FIG. 80 is a block diagram illustrating a construction forcorrelation filtering within the same receptive field in the same image,which is the correlation filtering between the ρ-axes.

A receptive field image is polar-transformed, and the polar-transformedreceptive field image is stored in the ρ-θ dual plane (hypercolumnmemory) HCM. Then, the polar-transformed receptive field image is cutout by the cut-out control portion for each of ρ-axes of differentorientations (348'), one-dimensional filtering processing is applied torespective outputs of the cut-out control portion to map onto dualplanes, respectively (352a'˜352n'), and correlation processing isperformed between data mapped onto the respective dual planes (3537).

According to the correlation filter of this embodiment, further finetexture analysis within a receptive field becomes possible, a simulationof the binocular stereopsis can be performed in the same way as thebinocular stereopsis for the same reason as the case of 3.5.2.2.1.Instead of the correlation between the ρ-axes as above, correlationbetween the θ-axes is also possible in the same way, and furthercorrelation on the (ρ,θ) plane is effective as two-dimensionalcorrelation.

3.5.2.3 Correlation between Different Color Images

Although the correlation filtering of intensity signals such asbrightness and the like of an input image are explained as above,further fine correlation filtering by detecting a delicate shade of acolor becomes possible when performing correlation filtering of FIG. 75by inputting color information as input data. Although various types ofcolor information which are processed can be input according to problemsto be solved, the basic color information is as follows.

3.5.2.3.1 Correlation in Three Primary Colors

The most basic color correlation is correlation in the three primarycolors. FIG. 81 is a block diagram illustrating a construction for colorcorrelation filtering between different images in the three primarycolors. A color image of one frame taken by a left camera CML isseparated through red, blue, and green filters (RFL, BFL, and GFL) intored, blue, and green images, and then, the red, blue, and green imagesare respectively divided into red, blue, and green receptive fieldimages IM_(R), IM_(B), and IM_(G) (360a˜360c), polar transformation isapplied to each receptive field image IM_(R), IM_(B), and IM_(G)(361a˜361c), and one-dimensional filtering processing is applied to theresult of the polar transformation to map onto dual planes (362a˜362c).Similarly, although not shown, a color image of one frame taken by aright camera is separated through red, blue, and green filters into red,blue, and green images and then, the red, blue, and green images arerespectively divided into red, blue, and green receptive field imagesIM_(R), IM_(B), and IM_(G), polar transformation is applied to eachreceptive field image IM_(R), IM_(B), and IM_(G), and one-dimensionalfiltering processing is applied to the result of the polartransformation to map onto dual planes. The, red correlation processing,blue correlation processing, and green correlation processing areperformed between red data mapped onto the dual planes, between bluedata mapped onto the dual planes, and between green data mapped onto thedual planes, respectively (363a˜363c).

By the above color correlation filtering, correlation parameters for thethree primary colors can be performed.

3.5.2.3.2 Correlation in Three Elements (Luminosity, Chroma, and Hue) ofcolor.

In another basic color correlation, the physical three primary colorsare transformed to brightness, chroma, and hue, which are amountssubjectively seen by human beings, and then correlation filtering isperformed. According to this filter, color correlation parameters, whichare psychologically arranged, can be measured.

FIG. 82 is a block diagram illustrating a construction for colorcorrelation filtering between different images in the three elements. Acolor image of one frame taken by a left camera CML is separated throughred, blue, and green filters into red, blue, and green images, and then,the color information by the three primary colors are transformed to thethree elements comprised of brightness IM_(Y), chroma IM_(S), and hueIM_(H) (359), the brightness, chroma, and hue images are respectivelydivided into brightness, chroma, and hue receptive field images IM_(V)', IM_(S) ', IM_(H) ' (360a'˜360c'), polar transformation is applied toeach receptive field image IM_(Y) ', IM_(S) ', and IM_(H) '(361a'˜361c'), and one-dimensional filtering processing is applied tothe result of the polar transformation to map onto dual planes(361a'˜362c'). Similarly, although not shown, a color image of one frametaken by a right camera is separated through red, blue, and greenfilters into red, blue, and green images, and then, the colorinformation by the three primary colors is transformed to the threeelements comprised of brightness, chroma, and hue. The brightness,chroma, and hue images are respectively divided into brightness, chroma,and hue receptive field images, polar transformation is applied to eachreceptive field image, and one-dimensional filtering processing isapplied to the result of the polar transformation to map onto dualplanes. Then, brightness correlation processing, chroma correlationprocessing, and hue correlation processing are performed betweenbrightness data mapped onto the dual planes, between chroma data mappedonto the dual planes, and between hue data mapped onto the dual planes,respectively (363a'˜363c').

3.5.2.3.3 Correlation in Color Difference Signals

For precisely measuring correlation parameters of a border portion ofcolors, such as a color at a transition stage of red and green, it iseffective to apply correlation processing to color difference signals.In particular, it is effective to use difference signals between thethree primary colors using complementary colors, instead of the threeprimary colors.

FIG. 83 is a block diagram illustrating a construction for colorcorrelation filtering between different images in the color differencesignals. A color image of one frame taken by a left camera CML isseparated through red, blue, and green filters into red, blue, and greenimages, and then, difference signals between red and green, green andblue, and blue and red are calculated (358a˜358c), the images of thedifference signals are respective difference signals IM_(R-G), IM_(G-B),IM_(B-R), polar transformation is applied to each receptive field imageIM_(R-G), IM_(G-B), IM_(B-R) (361a"˜361c"), and one-dimensionalfiltering processing is applied to the result of the polartransformation to map onto dual planes (361a"˜362c"). Similarly,although not shown, a color image of one frame taken by a right camerais separated through red, blue, and green filters into red, blue, andgreen images, and then, difference signals between red and green, greenand blue, and blue and red are calculated, the images of the differencesignals are respectively divided into receptive field images of therespective difference signals, polar transformation is applied to eachreceptive field image, and one-dimensional filtering processing isapplied to the result of the polar transformation to map onto dualplanes. Then, red-green correlation processing, green-blue correlationprocessing, and blue-red correlation processing are performed betweenred-green difference data mapped onto the dual planes, betweengreen-blue difference data mapped onto the dual planes, and betweenblue-red difference data mapped onto the dual planes, respectively.

Alternatively, instead of the difference, division or difference aftertransformation to a logarithm, is also effective. In addition, thedifference or division may be performed after the polar transformationor the one-dimensional filter. The division is effective when theintensities of three primary colors greatly vary, and the differenceafter transformation to a logarithm is effective to emphasize colordifference in the dark portion.

Although the color correlation between different images is explainedabove, the color correlation method can be combined with each typecorrelation method explained in the above sub-sections 3.5.2.1 and3.5.2.2, and the correlation filtering such as the binocular stereopsis,the grazing of a moving object, texture analysis, and the like, can befurther surely performed.

3.5.3 Operating Direction of Correlation

Although, in the example of the binocular stereopsis (see FIG. 70),correlation is performed from the data in the ρ-direction, where theorientation θ in the dual plane is the same, the correlation is notnecessarily limited to the ρ-axis, and the following generalizationwithin the ρ-axis is possible.

3.5.3.1 Correlation in ρ-Axis Direction

3.5.3.1.1 ρ-Axis Correlation of Same θ

This is the case of the above binocular stereopsis and the like. Themeaning of the "ρ-axis correlation of the same θ" is, as explained withreference to FIG. 69, to precisely measure the offset between theparallel lines in both images by the correlation method. FIG. 84 is ablock diagram illustrating a construction for the ρ-axis correlationfiltering for the same θ. Each of input data A and B is divided intoreceptive field images (350a, 350b), polar transformation processing isapplied to each receptive field image (351a, 351b), one-dimensionalfiltering processing is applied to the result of the polartransformation to map onto dual planes (351a, 352b), data in theρ-direction is selected for each of θ₀, θ₁, . . . from each dual plane(354a₀, 354a₁, . . . ; 354b₀, 354b₁, . . . ), and correlation isperformed between data of the same θ value (353₀, 353₁, . . . ).Although FIG. 84 indicates the construction wherein the correlationcalculations for the respective θ values are performed in parallel, thecorrelation calculations for the respective θ values may be performed inturn by one correlation calculation portion. In the latter case, theconstruction is the same as the block diagram of FIG. 76A.

3.5.3.1.2 ρ-Axis Correlation for Different θ's

In this correlation, a correlation parameter between tangential lineseach having a different orientation is measured. By this correlation,extraction of a moving tangential line the orientation of which variesduring the movement is possible, in addition to the correlation betweenparallel lines in 3.5.3.1.1. This correlation is expected to be appliedto cars running on a bad road. FIG. 85 is a block diagram illustratingthe construction for the ρ-axis correlation filtering between differentθ's.

Each of input data A and B is divided into receptive field images (350a,350b), polar transformation processing is applied to each receptivefield image (351a, 351b), one-dimensional filtering processing isapplied to the result of the polar transformation to map onto dualplanes (352a, 352b), data in the ρ-direction is selected for each ofθ_(i), θ_(i+1), . . . ; θ_(j), θ_(j+1), . . . from each dual plane(354a₀ ', 354a₁ ', . . . ; 354b₀ ', 354b₁ ', . . . ), and correlation isperformed between data of the different θ values (353₀ ', 353₁ ', . . .). The correlation can be performed, for example, between θ_(i) andθ_(i+1), θ_(i-1) and θ_(i) (adjacent angles).

3.5.3.2 Correlation in θ-Direction

The meaning of the correlation parameter in the ρ-direction for the sameρ is explained with reference to FIGS. 86A, 86B, 86C and 86D. In the θcorrelation for ρ=0 (FIGS. 86A and 86B), a rotation amount of atangential line SL passing through a center C of a receptive field isextracted. In the θ correlation for ρ≠0, a rotation amount of atangential line SL contacting a circle CIR of a radius ρ and having thecenter of the circle at the center of the receptive field, can bedetected (FIGS. 86C and 86D).

FIG. 87 is a block diagram illustrating a construction for the θ-axiscorrelation filtering for the same ρ. Each of input data A and B isdivided into receptive field images (350a, 350b), polar transformationprocessing is applied to each receptive field image (351a, 351b),one-dimensional filtering processing is applied to the result of thepolar transformation to map onto dual planes (351a, 352b), data in theθ-direction is selected for each of ρ₀, ρ₁, . . . from each dual plane(355a₀, 355a₁, . . . ; 355b₀, 355b₁ . . . ), and correlation isperformed between data of the same θ value (353₀ ", 353₁ " . . . ).Although FIG. 87 indicates the construction wherein the correlationcalculation for the respective ρ values is performed in parallel, thecorrelation calculation for the respective ρ values may be performed inturn by one correlation calculation portion.

3.5.3.3 Correlation on (ρ,θ) Plane

Although one-dimensional correlation on the ρ-axis or θ-axis isexplained, two-dimensional correlation on the (ρ,θ) plane is alsopossible. According to this correlation, precise correlation of atangential line moving while varying their orientations is possible.

FIG. 88 is a block diagram illustrating a construction for thecorrelation filtering on the (ρ,θ) plane. Each of input data A and B isdivided into receptive field images (350a, 350b), polar transformationprocessing is applied to each receptive field image (351a, 351b),one-dimensional filtering processing is applied to the result of thepolar transformation to map onto dual planes (352a, 352b), andcorrelation processing for two-dimensional shift amount (σ₁,σ₂) isperformed on each dual plane (3538). When data on each dual plane isdenoted by a(ρ,θ) and b(ρ,θ), and shift amounts on the dual planes inthe ρ-direction and the θ-direction are denoted by σ₁,σ₂, correlationamount C(ρ,θ,σ₁,σ₂) is expressed by the following equation.

    C(ρ, θ,σ.sub.1,σ.sub.2)=a(ρ,θ)·b(ρ+.sigma..sub.1, θ+σ.sub.2)                         (102)

3.5.4 Correlation Parameter

Although, to simplify the explanation, in the examples of the binocularstereopsis, the method for obtaining corresponding tangential linesbased on the correlation amount (correlation parameter) according to theequation (101) is explained, various types of correlation parameters canbe defined as explained below, and correlation processing characteristicto the respective types of correlation parameters can be performed.

3.5.4.1 Basic Correlation Parameter

As indicated below, generally, the basic correlation parameter can beexpressed by three-dimensional representation such as C(ρ,θ,σ), andthree-dimensional space of ρ, θ, and σ is as indicated in FIG. 89. InFIG. 89, τ denotes a shift amount (movement velocity) between data,which is introduced in the correlation between images at differenttimes, and is explained later. When a point (ρ_(P),θ_(P),σ_(P)) in thethree-dimensional space, wherein the correlation parameter C(ρ,θ,σ) is alocal maximum, is obtained, determination of correspondence in thebinocular stereopsis, determination of correspondence of a pair ofmoving tangential lines, and the like, are performed. In addition,locations, orientations, parallaxes, and velocities of the respectivetangential lines can be measured based on the respective coordinatevalues (values of the element parameters) which give the local maximum.In the two-dimensional correlation on the (ρ,θ) plane, correlationparameter is a parameter C(ρ,θ,σ₁,σ₂) in the four-dimensional space.

3.5.4.1.1 Spatial Correlation Parameter

In the case where images on which the correlation is performed, arespatially different images or spatially different receptive fields (asthe cases explained in the sub-sections 3.5.2.1.1 and 3.5.2.2), acorrelation amount indicating a degree of correspondence such as aparallax of corresponding tangential lines, can be measured based on alocal maximum of a correlation parameter as explained below.

(1) Correlation Parameter C(ρ,θ,σ) in the ρ Direction

The correlation parameter C(ρ,θ,σ) in the ρ direction is a parameterindicating a degree of spatial correlation of a group of parallel lines.The correlation parameter in the aforementioned binocular stereopsis isan example. When ρ_(P), θ_(P), and σ_(P) where C(ρ,θ,σ) is a localmaximum, are obtained, a tangential line is determined from a group ofparallel lines, the location, orientation, and binocular parallax of thetangential line are quantitatively measured. When data on each dualplane is denoted by a(ρ,θ) and b(ρ,θ), and a shift amount on the dualplanes in the ρ-direction is denoted by σ, two-dimensional correlationparameter C(ρ,θ,σ₁,σ₂) is expressed by the following equation.

    C(ρ,θ,σ)=a(ρ,θ)·b(ρ+σ,θ)

(2) Correlation Parameter Cθ(ρ,θ,σ) in θ Direction

The correlation parameter Cθ(ρ,θ,σ) in the θ direction is a parameterindicating a degree of spatial correlation of a group of tangentiallines contacting a circle CIR of a radius ρ and having the center of thecircle at the center of the receptive field. In the θ correlationprocessing for ρ=0, corresponding tangential lines are determined amonga group of the radial lines based on ρ_(P), θ_(P), and σ_(P) which makeCθ(ρ,θ,σ) a local maximum, and the location, orientation, and variationamount of the orientation of the tangential line are quantitativelymeasured. When a shift amount in the θ-direction is denoted by σ, thecorrelation parameter is expressed by the following equation.

    Cθ(ρ,θ,σ)=a(ρ,θ)·b(ρ,θ+.sigma.)                                                    (103)

(3) Correlation Parameter C(ρ,θ,σ₁,σ₂) in (ρ,θ) Plane

The two-dimensional correlation parameter C(ρ,θ,σ₁,σ₂) in the (ρ,θ)plane is a parameter indicating a degree of spatial correlation of agroup of tangential lines moving while varying their orientations, andthereby the degree of the two-dimensional correlation in the dual planescan be evaluated in more detail compared with the above one-dimensionalcorrelation parameter. The correlation parameter is given by theequation (102), i.e., by the following equation.

    C(ρ,θ,σ.sub.1,σ.sub.2)=a(ρ,θ)·b(.rho.+σ.sub.1,θ+σ.sub.2)

3.5.4.1.2 Time Correlation Parameter

In the case images on which the correlation is performed, are images atdifferent times or receptive fields at different times (as the caseexplained in the sub-section 3.5.2.1.2), a moving direction and amovement velocity of a moving tangential line can be measured based on alocal maximum of a correlation parameter.

(1) Correlation Parameter C(ρ,θ,τ) in ρ-Direction

The correlation parameter C(ρ,θ,τ) in the ρ-direction is a parameterindicating a degree of correlation of a group of parallel linesregarding time. The correlation parameter in the embodiment explainedlater, i.e., in the case of pursuing a moving tangential line, is anexample of this case. Based on ρ_(P), θ_(P), and τ_(P) which makeC(ρ,θ,τ) a local maximum, a location, an orientation, and a velocity ofa translating tangential line can be measured quantitatively. When thetranslating velocity in the ρ-direction is denoted by τ, the correlationparameter is expressed by the following equation

    C(ρ,θ,τ)=a(ρ,θ)·b(ρ+τ,θ).(104)

When defining as a(ρ,θ)=a_(t) (ρ,θ), b(ρ+τ,θ)=a_(t+DELTA) (ρ+τ,θ), thecorrelation parameter is given by the following equation.

    C(ρ,θ,τ)=a.sub.t (ρ,θ)·a.sub.t+DELTA (ρ+τ,θ)                                     (104)'

Namely, in 3534 of FIG. 77, correlation processing of the equation(104)' is performed.

When DELTA in 349 of FIG. 77 denotes the delay time, V denotes atranslating velocity of a tangential line, Δρ denotes a resolution inthe ρ direction, and τ_(P) is a value of τ at which C(ρ,θ,τ) becomes alocal maximum, the translating velocity V is obtained by

    V=(τ.sub.P ·Δρ)/DELTA.

The above operation corresponds to a modeled function of the"directional selective simple cell" in the visual field of the cerebrum.The directional selective simple cell has a characteristic of respondingto a stimulus moving in a specific direction, but not responding to astimulus moving in the opposite direction. This function is realized bylimiting the range of τ in the above equation to positive or negative.This characteristic is a reflection of the limitation that the organicnerve cannot transmit positive and negative information simultaneously.Precisely, the above directional selective simple cell is the B-typedirectional selective cell, which is a kind of the simple cell, and hasa response independent of brightness/darkness of a stimulus.

The above time correlation parameter can be calculated according to thefollowing equation (104)", instead of the equation (104).

    C(ρ,θ,τ)=a(ρ-τ,θ)·b(ρ+τ,.theta.)                                                         (104)"

Hereinafter, it is assumed that the asymmetric-type correlationcalculation is performed according to the equation (104).

(2) Correlation Parameter Cθ(ρ,θ,τ) in θ-Direction

The correlation parameter Cθ(ρ,θ,τ) in the θ-direction is a parameterindicating a degree of time correlation of a group of tangential linescontacting a circle of a radius ρ and having the center of the circle atthe center of the receptive field. In the θ correlation processing forρ=0, corresponding tangential lines are determined among a group of theradial lines based on ρ_(P), θ_(P), and σ_(P) which make Cθ(ρ,θ,σ) alocal maximum, and the location, orientation, and rotation velocity ofthe orientation of the tangential line are quantitatively measured. Whena movement velocity in the θ-direction is denoted by τ, the correlationparameter is expressed by the following equation.

    Cθ(ρ,θ,τ)=a(ρ,θ)·b(ρ,θ+.tau.)                                                        (105)

When defining as a(ρ,θ)=a_(t) (ρ,θ), and B(ρ,θ+τ)=a_(t+DELTA) (ρ,θ+τ) ,the correlation parameter is given by the following equation.

    Cθ(ρ,θ,τ)=a.sub.t (ρ,θ)·a.sub.t+DELTA (ρ,θ+τ)(105)'

Namely, in 3534 of FIG. 77, correlation processing according to theequation (105)' is performed.

When the delay time is denoted by DELTA, and τ_(P) is a value of τ atwhich Cθ(ρ,θ,τ) becomes a local maximum, the rotation velocity ω isobtained as

    ω=(τρ·Δρ)/DELTA.

In the visual cortex of the cerebrum, there are cells for detecting therotation, and the above operation corresponds to a modeled function ofthe cells.

(3) Correlation Parameter C(ρ,θ,τ₁,τ₂) in (ρ,θ) Plane

The two-dimensional correlation parameter C(ρ,θ,τ₁,τ₂) in the (ρ,θ)plane is a parameter indicating a degree of time correlation of a groupof tangential lines moving while varying their orientations. Based onρ_(P), θ_(P), τ_(1P), τ_(2P) which make C(ρ,θ,τ₁,τ₂) a local maximum, alocation, an orientation, a movement velocity, and a rotation velocityof the tangential line passing through a center of a receptive field,can be measured in more detail compared with the one-dimensionalcorrelation parameter. The correlation parameter is given by thefollowing equation.

    C(ρ,θ,τ.sub.1,τ.sub.2)=a(ρ,θ)·b(ρ+τ.sub.1,θ+τ.sub.2)                           (106)

Namely, in 3534 of FIG. 77, the correlation processing according to theequation (106) is performed. When defining a(ρ,θ)=a_(t) (ρ,θ) andb(ρ+τ₁,θ+τ₂)=a_(t+DELTA) (ρ+τ₁,θ+τ₂), the correlation parameter is givenby the following equation.

    C(ρ,θ,τ.sub.1,τ.sub.2)=a.sub.t (ρ,θ)·a.sub.t+DELTA (ρ+τ.sub.1,θ+τ.sub.2)                   (106)'

Namely, in 3534 of FIG. 77, the correlation processing according to theequation (106)' is performed. When the delay time is denoted by DELTA in349 of FIG. 77 and τ_(1P) and τ_(2P) are values of τ₁ and τ₂,respectively, at which C(ρ,θ,τ₁,τ₂) becomes a local maximum, thetranslation velocity V and the rotation velocity ω are obtained as

    V=(τ.sub.1P ·Δρ)/DELTA,

and

    ω=(τ.sub.2P ·Δρ)/DELTA.

3.5.4.2 Projection of Basic Correlation Parameter

Although, by the above basic correlation parameters, all the correlationparameters including a location (ρ_(P)), an orientation (θ_(P)), andbinocular parallax (σ_(P)) or movement velocity (τ_(P)) of a tangentialline, can be determined, the dimensions of a storage portion for storingcorrelation parameter is required to be three or four.

Hereinafter, definitions and characteristics of correlation parametersfor reducing the capacity of the correlation parameter storage portionby suppressing the number of parameters for correlation decision asneeded, are explained below.

Although the explanation is given for spatial correlation parameterC(ρ,θ,σ) and time correlation parameter C(ρ,θ,τ), two-dimensionalcorrelation parameters C(ρ,θ,σ₁,σ₂) and C(ρ,θ,τ₁,τ₂) on the (ρ,θ) planecan also be defined, and correlation measurement in more detail than theone-dimensional correlation parameter is possible.

3.5.4.2.1 Projection in σ or τ Direction

In the case where the location ρ and the orientation θ of correspondingtangential lines are important, and the direct measurement of thebinocular parallax σ or movement velocity τ is not required, the amountC_(PRJ) -σ(ρ,θ) or C_(PRJ-)τ (ρ,θ) generated by accumulating the basiccorrelation parameter C(ρ,θ,σ) or C(ρ,θ,τ) in the σ- or τ-direction, andexpressed by the following equation is effective, and thereby thecorrelation parameter storage portion is reduced to a two-dimensionalmemory of the same as the dimensions of the input dual plane. ##EQU20##

FIG. 90 is a block diagram illustrating a construction for the filteringprocess wherein the basic correlation parameter C(ρ,θ,σ) is projected inaccordance with the equation (107). Each of input data A and B isdivided into receptive field images (350a, 350b), polar transformationprocessing is applied to each receptive field image (351a, 351b),one-dimensional filtering processing is applied to the result of thepolar transformation to map onto dual planes (352a, 352b), correlationprocessing is performed on data in the respective dual planes accordingto the equation (101) (3539), correlation parameter C(ρ,θ,σ) obtained bycorrelation processing is accumulated according to the equation (107)(371), and the accumulated result is stored in the ρ-θ correlationparameter memory (372).

FIG. 91 is a flowchart of processing of calculating correlationparameters, and projecting the same in the σ direction.

When starting the correlation calculation, 0→θ, 0→ρ, 0→σ (3201˜3203).Next, correlation parameters C(ρ,θ,σ) are calculated according to theequation (101) (3204), the correlation parameters (107) are accumulatedaccording to the equation (3205). Then, a is incremented, it isdetermined whether or not σ>σ_(max) (3206, 3207). When it is determinedthat σ≦σ_(max), the operation goes to step 3204, and the followingprocessing is repeated. When it is determined that σ>σ_(max), theaccumulated value is stored at the matrix intersecting point, indicatedby ρ and θ, in the correlation parameter storage portion (3208).Thereafter, ρ is incremented, and it is determined whether or notρ>ρ_(max) (3209, 3210). When it is determined that ρ≦ρ_(max), theoperation goes to step 3203, and the following processing is repeated.When it is determined that ρ>ρ_(max), θ is incremented, and it isdetermined whether or not θ>θ_(max) (3211, 3212). When it is determinedthat θ≦θ_(max), the operation goes to step 3202, and the followingprocessing is repeated. When it is determined that θ>θ_(max), thecorrelation calculation and the projection processing are completed

3.5.4.2.2 Projection in ρ Direction

In the case where the binocular parallax σ (movement velocity τ) and theorientation θ of corresponding tangential lines are important, and thedirect measurement of the location ρ is not required, the amountC_(PRJ-)ρ (ρ,θ) or C_(PRJ-)ρ (ρ,τ) generated by accumulating the basiccorrelation parameter C(ρ,θ,σ) in the ρ-direction, and expressed by thefollowing equation is effective, and thereby the correlation parameterstorage portion is reduced to two dimensions which are the same as thedimensions of the input dual plane.

    C.sub.PRJ-ρ (θ,σ)=ΣC(ρ,θ,σ)(109)

(where ρ=0, 1, 2, . . . ρ_(max))

    C.sub.PRJ-ρ (θ,τ)=ΣC(ρ,θ,τ)(110)

(where ρ=0, 1, 2, . . . ρ_(max))

These correlation parameters are effective to detect contour linesmoving in a field of view, an orientation θ and a movement velocity τ ofa moving contour tangential line, can be measured independent of thelocation ρ of the tangential line.

Cells operating as above exist in the hypercolumn in the primary visualcortex of the cerebrum. The cells are known as "velocity selectivecomplex cells", which are strongly excited while capturing when astraight line in a field of view translates, and does not output asignal when the straight line stops. The equation (110) plays animportant role in "measurement of an optical flow", and the equation(109) plays an important role in "binocular stereopsis from random-dots"explained later.

3.5.4.2.3 Projection in θ Direction

In the case the binocular parallax σ (movement velocity τ) and thelocation ρ of corresponding tangential lines are important, and thedirect measurement of the orientation θ is not required, the amountC_(PRJ-)θ (ρ,σ) or C_(PRJ-)θ (ρ,τ) generated by accumulating the basiccorrelation parameter C(ρ,θ,σ) in the θ-direction, and expressed by thefollowing equation is effective, and thereby the correlation parameterstorage portion is reduced to two dimensions which are the same as thedimensions of the input dual plane.

    C.sub.PRJ-θ (ρ,σ)=ΣC(ρ,θ,σ)(111)

(where θ=0, 1, 2, . . . θ_(max))

    C.sub.PRJ-θ (ρ,τ)=ΣC(ρ,θ,τ)(112)

(where θ=0, 1, 2, . . . θ_(max))

The above accumulated correlation parameter is effective to pursuing amoving contour line. The distance ρ of the contour line from the centerof the receptive field and the movement velocity τ of the contour linecan be measured independently of the orientation θ of the line. Afunction similar to the above is realized in the cells existing in theretina of a frog for pursuing game such as a fly.

3.5.4.2.4 Projection in ρσ- or ρτ-Direction

In the case the orientation θ of corresponding tangential lines isimportant, and the direct measurement of the binocular parallax σ (or amovement velocity τ) and the location ρ is not required, the amountC_(PRJ-)ρσ (θ) or C_(PRJ-)ρτ (θ) generated by accumulating the basiccorrelation parameter C(ρ,θ,σ) in the ρσ-direction (or ρτ-direction),and expressed by the following equation is effective, and thereby thecorrelation parameter storage portion is reduced to one dimension.

    C.sub.PRJ-ρσ (θ)=ΣΣC(ρ,θ,σ)(113)

(where ρ=0, 1, 2, . . . ρ_(max), σ=0, 1, 2, . . . σ_(max))

    C.sub.PRJ-ρτ (θ)=ΣΣC(ρ,θ,τ)(114)

(where ρ=0, 1, 2, . . . ρ_(max), τ=0, 1, 2, . . . τ_(max))

The above accumulated correlation parameter is effective to detect thata contour line in an orientation moves in a receptive field. Theorientation θ of a moving contour line can be measured independently ofthe location and movement velocity of the contour line.

3.5.4.2.5 Projection in ρθ Direction

In the case the parallax a (or a movement velocity τ) of correspondingtangential lines is important, and the direct measurement of thelocation ρ and the orientation θ is not required, the amount C_(PRJ-)ρθ(σ) or C_(PRJ-)ρθ (τ) generated by accumulating the basic correlationparameter C(ρ,θ,σ) in the ρθ-direction, and expressed by the followingequation is effective, and thereby the correlation parameter storageportion is reduced to one dimension.

    C.sub.PRJ-ρθ (σ)=ΣΣC(ρ,θ,σ)(115)

(where ρ=0, 1, 2, . . . ρ_(max),θ=0, 1, 2, . . . θ_(max))

    C.sub.PRJ-ρθ (τ)=ΣΣC(ρ,θ,τ)(116)

(where ρ=0, 1, 2, . . . ρ_(max),θ=0, 1, 2, . . . θ_(max))

The above accumulated correlation parameter is effective: to "determinewhether or not there are contour lines seen by the right and left eyesin the receptive field, and corresponding to each other, and detect thebinocular parallax"; or to "determine whether or not there is a contourline moving in a receptive field, and detect the movement velocity ofthe contour line". In the case of the equation (116), the velocity τ ofthe moving contour line can be measured independently of the locationand orientation of the contour line.

3.5.4.2.6 Projection in θρσ- or θρτ-Direction

In the case where the only important thing to know is to determinewhether or not corresponding contour lines exist in the receptive fieldsthe parallax ρ (movement velocity τ) of a corresponding tangential linesis important, and the direct measurement of the location ρ and theorientation θ is not required, the amount C_(PRJ-)θρσ or C_(PRJ-)θρτgenerated by accumulating the basic correlation parameter C(ρ,θ,σ) inthe ρθσ-direction (or ρθτ-direction), and expressed by the followingequation is effective, and thereby the correlation parameter storageportion is reduced to zero dimensions corresponding to only one cell.

    C.sub.PRJ-θρσ =ΣΣΣC(ρ,θ,σ)(117)

(where ρ=0, 1, 2, . . . ρ_(max),θ=0, 1, 2, . . . θ_(max), σ=0, 1, 2, . .. σ_(max))

    C.sub.PRJ-θστ =ΣΣΣC(ρ,θ,τ)(118)

(where ρ=0, 1, 2, . . . ρ_(max),θ=0, 1, 2, . . . θ_(max), τ=0, 1, 2, . .. τ_(max))

For example, in the case of the equation (118), the above accumulatedcorrelation parameter is effective to detect existence only of a contourline moving in a receptive field, and the detection can be performedindependently of the location, orientation, and the movement velocity ofthe contour line. A function similar to the above is realized in thecells commonly existing in the retinas of lower animals such as a frogfor quickly detecting that game such as a fly is in a receptive field,with a small number of cells. The intelligence of nature wherein aprocessing system is constructed with an extremely small number ofcells, can be appreciated. From the point of view of engineering, theabove technique is effective to recognize an outline of information onmovement for each receptive field, for example, in an unmanned vehiclefor surveying a planet.

3.5.4.2.7 Projection in Oblique Direction in (ρ,τ) or (ρ,σ) Plane

In the above embodiments, the basic correlation parameters C(ρ,θ,σ),C(ρ,θ,τ), and the like are projected directly in the directions of theelement parameters, i.e., directly in the directions of the location ρ,the orientation θ, the binocular parallax σ, and the movement velocity τof the contour line, to reduce the capacity of the correlation parameterstorage portion. Here, a method for reducing the capacity of thecorrelation parameter storage portion with maintaining thecharacteristics of the above element parameters, is explained.

When the basic correlation parameters C(ρ,θ,σ) and C(ρ,θ,τ) areprojected in an arbitrary oblique direction in the (ρ,τ) or (ρ,σ) planeof FIG. 89, the correlation parameter storage portion can be reduced totwo dimensions while maintaining the characteristics of ρ, τ, and σ.This oblique projection is explained for an example wherein theprojection is performed in the 45° direction in the (ρ,σ) plane. When anaxis perpendicular to the projection direction is denoted by ξ, thecorrelation parameter C_(PRJ-45)σ (θ,ξ) projected in the 45° direction,is expressed by the following equation. ##EQU21## When θ_(P) and ξ_(P)where the correlation parameter C_(PRJ-45)σ (θ,ξ) is a local maximum, isobtained, the orientation θ_(P) of the corresponding contour lines andthe "parameter ξ_(P) which is generated by combining the location ρ andparallax σ" are calculated.

3.5.4.3 Natural Filter of Convolution Filter C₁ (ρ,θ)

3.5.4.3.1 When output data on the dual planes are denoted by a(ρ,θ) andb(ρ,θ), the following projection correlation parameter C₁ (ρ,θ)constitutes an interesting filter. ##EQU22##

On the other hand, the convolution calculation output C₂ (ρ,θ) by theone-dimensional filter in the aforementioned second aspect of thepresent invention is given by the following equation, where theone-dimensional filter function is denoted by g(ρ). ##EQU23##

Comparing the above two equations, the filter function g(ρ) in theconvolution output C₂ (ρ,θ) is replaced with the correlation data b(i,θ)in the correlation parameter C₁ (ρ,θ). Therefore, the correlationparameter C₁ (ρ,θ) can be considered to be a "natural filtering" resultwherein convolution calculation is performed with natural data b(i,θ)instead of the artificial filter function g(ρ).

The functions of the above two types of filtering are considered below.When the filter function g(ρ) is fixed, the "artificial filtering"according to the equation (121) can be adopted, where the operation suchas differentiation is determined in advance. However, when extracting a"characteristic feature seen in the same way by the right and lefteyes", or pursuing "the same characteristic feature as the previousimage", the convolution calculation with the natural data is necessary.Namely, these operations are impossible by the "artificial filtering".On the other hand, by the "natural filtering" according to the equation(120), precise filtering can be performed in the cases the filterpatterns cannot be determined in advance.

FIG. 92 is a diagram illustrating the construction of the naturalfiltering. In FIG. 92, reference numeral 381 denotes a receptive fielddivision portion which divides an input image (of one frame) intoreceptive fields to output the divided images, 382 denotes a polartransformation portion which applies predetermined polar transformationto the receptive field image, 383 denotes a ρ-θ dual plane (hypercolumnmemory) for storing a result a(ρ,θ) of the polar transformation, and 384denotes a natural filter according to the present invention. The naturalfilter comprising: a receptive field portion 384a for cutting out thenatural filter data into receptive fields; a polar transformationportion 384b for applying polar transformation processing to eachreceptive field image; a ρ-θ dual plane 384c for storing a result b(ρ,θ)of the polar transformation, and a multiplication portion 384d and anaccumulation portion 384e for performing the calculation according tothe equation (120).

The multiplication portion 384d performs the multiplication σ(ρ+i,θ)·b(i,θ), and outputs the multiplied result to the accumulation portion384e, where a(ρ+i,θ) and b(i,θ) (initial values ρ, θ, and i are 0) eachdenote data output from the dual planes 383 and 384c. Then, i isincremented, and a similar multiplication is repeated for the next dataa(ρ+i,θ) and b(i,θ) until i>i_(max) (where i_(max) is a filter width,and is preset appropriately, for example, i_(max) =ρ_(max)). Theaccumulation portion 384e accumulates the results of multiplication, andstores the accumulated result in the ρ-θ plane memory (not shown) when ireaches i>i_(max). Hereinafter, the sum of products is calculated forall values of (ρ,θ), and natural filtering processing is completed.

3.5.4.3.2 Application to Binocular Stereopsis

Although the correspondence of fine contour lines can be obtained inthree dimensions by the basic correlation parameter C(ρ,θ,σ), thecorrespondence of contour lines can be obtained in the two-dimensionalplanes (ρ,θ).

Concretely, by obtaining a local maximum of the aforementioned C₁ (ρ,θ),the location ρ and the orientation θ of the corresponding contour linescan be determined by two eyes independently of the parallax σ.

3.5.4.3.3 Application to Pursuit of Movement Contour Line

Although natural filters between spatially different receptive fieldsare explained above, the natural filter is effective regarding time.When input data at different times are denoted by a_(t) (ρ,θ) anda_(t+DELTA) (ρ,θ), respectively, the time correlation parameter C_(time)(ρ,θ) is expressed by the following equation. ##EQU24##

As explained above, the location, the orientation, and the movementvelocity of moving contour lines can be obtained in three-dimensionalspace by the basic time correlation parameter C(ρ,θ,σ). However,according to the above natural filter, the location ρ and theorientation θ of moving contour lines can be determined independently ofthe movement velocity τ, by obtaining a local maximum of C_(time) (ρ,θ).

3.5.4.4 Different Types of Basic Correlation

Although the basic correlation parameters are defined by multiplicationas

    C(ρ,θ,σ)=a(ρ,θ)·b(ρ+σ,θ),

and

    C(ρ,θ,τ)=a.sub.t ( ρ,θ)·a.sub.t+DELTA (ρ+τ,θ),

the multiplication may be replaced with addition to obtain a similareffect. In this case, linearity between signals is assured, and it isknown that the "linear simple cell" in the visual cortex of the cerebrumhas a similar function. Further, when the above multiplication in theabove equations is replaced with subtraction, the binocular stereopsisand pursuit of an object having an obscure contour can be performedbased on gradual variations of brightness, hue, and the like.

3.5.4.4.1 Spatial Correlation

Explaining for the case of the ρ-direction correlation, the basiccorrelation parameter is given by

    C'(ρ,θ,σ)=a(ρ,θ)-b(ρ+σ,θ).(123)

In the above correlation, C'(ρ,θ,σ) is zero where input data are equal.Due to this characteristic, the binocular stereopsis and pursuit of anobject having an obscure contour becomes possible. For example, whenseeing a large round cylinder, no clear contour will be seen except atboth their ends. However, brightness will gradually vary according to arelationship with the incident angles. By searching for a location wherebrightness by the right and left eyes is the same based on thecharacteristic that C'(ρ,θ,σ) is zero where input data are equal,correspondence between two eyes can be obtained, and a surface conditionof the cylinder can be measured by stereopsis.

FIG. 93 is a flowchart of correlation processing of the difference type,where the whole process basically corresponds to the basic block diagramof FIG. 75 except that the calculation of correlation parameters isperformed according to the equation (123). When starting the correlationcalculation, 0→θ, 0→ρ, 0→σ (3301˜3303). Next, correlation parameters arecalculated according to the equation (123), and the calculatedcorrelation parameters are stored in the correlation parameter storageportion (3304, 3305). Then, σ is incremented, and it is determinedwhether or not σ>σ_(max) (3306, 3307). When it is determined thatσ≦σ_(max), the operation goes to step 3304 to repeat the followingoperations. When it is determined that σ>σ_(max), ρ is incremented, andit is determined whether or not ρ>ρ_(max) (3308, 3309). When it isdetermined that ρ≦ρ_(max), the operation goes to step 3303 to repeat thefollowing operations. When it is determined that ρ>ρ_(max), θ isincremented, and it is determined whether or not θ>θ_(max) (3310, 3311).When it is determined that θ≦θ_(max), the operation goes to step 3302 torepeat the following operations. When it is determined that θ>θ_(max),the processing of correlation calculation is completed.

3.5.4.4.2 Time Correlation

Explaining for the case of ρ direction, the basic correlation parameteris given by the following equation.

    C'(ρ,θ,τ)=a(ρ,θ)-b(ρ+τ,θ)(123)'

When defining as a(ρ,θ)=a_(t) (ρ,θ) and b(ρ+τ,θ)=a_(t+DELTA) (ρ+τ,θ),the correlation parameter is given by the following equation.

    C'(ρ,θ,τ)=a.sub.t (ρ,θ)-a.sub.t+DELTA (ρ+τ,θ)                                     (123)"

In this correlation, C'(ρ,θ,τ) is zero where input data are equal. Dueto this characteristic, the pursuit of an object having an obscurecontour becomes possible. According to a method similar to the abovecase of 3.5.4.4.1, by searching for a location where brightness by theright and left eyes is the same based on the characteristic thatC'(ρ,θ,σ) is zero where input data are equal, a stable pursuit of aportion where a clear contour line cannot be detected, such as thesurface of a cylinder or a portion of a human face, becomes possible.Cells having a similar function are "directional selective simple cells"which belong to "linear response-type cells".

3.6 Embodiment of Concrete System

3.6.1 Binocular Stereopsis

As a concrete example of the case of spatially different images, asystem and a simulation example of the binocular stereopsis areexplained. Characteristic features of tangential lines include a line, agap, and an edge. The "line" is a luminous band, the "edge" is a borderline between a luminous portion and a dark portion, and the "gap" isinversion of a line, i.e., a dark narrow band. Although thecorrespondence of "lines" by the binocular stereopsis is explained forexplaining the principle of the correlation filter in the sub-section3.4, the binocular stereopsis is explained here.

3.6.1.1 Various Filter Construction

Although, in the explanation of the second aspect of the presentinvention, the line extraction filter, the edge extraction filter, andthe gap extraction filter are disclosed, by using these filters, thebinocular stereopsis of lines and gaps, and the binocular stereopsis ofedges becomes possible. That is, the filter indicated in FIGS. 50A, 50Band 50C can be used as a line extraction filter. The filter indicated inFIGS. 52A, 52B, 52C and 52D can be used as an edge extraction filter.The filter indicated in FIGS. 57A, 57B and 57C can be used as a gapextraction filter.

3.6.1.2 Binocular Stereopsis of Line and Gap

3.6.1.2.1 Binocular Stereopsis of Line and Gap by One-Dimensional Filter

When replacing the process of "polar transformation+one-dimensionalfilter processing" according to the basic block diagram of FIG. 75, withthe procedure of "receptive field division+polartransformation+one-dimensional Gaussian filter (or one-dimensionalGaussian filter+polarity inversion) processing" indicated in FIGS. 50Band 57B, respectively, and inputting images of the right and left eyes,the binocular stereopsis of lines and gaps becomes possible. Thesimulation result of the above process is already explained in FIGS.72A˜74. The sharp peak of C(ρ,θ,σ) in FIG. 74 corresponds tocorresponding points of perpendicular lines, and precise measurementthat the binocular parallax is equal to five pixels, is performed basedon the values on the σ-axis.

3.6.1.2.2 Binocular Stereopsis of Lines and Gaps by Two-DimensionalFilter (Binocular Stereopsis in the Visual Cortex of the Cerebrum)

When replacing the process of "polar transformation+one-dimensionalfilter processing" according to the basic block diagram of FIG. 75, withthe procedure of "two-dimensional Gaussian filter+receptive fielddivision+polar transformation (or polar transformation+polarityinversion) processing" indicated in FIGS. 50C and 57C, respectively, andinputting images of the right and left eyes, the binocular stereopsis oflines, and gaps becomes possible. In this system, two-dimensional seconddifferentiation processing (two-dimensional convolution filterprocessing) is applied to the input images of the right and left eyes bythe two-dimensional gas filter. Next, the filtered input images arerespectively divided into receptive field images, and polartransformation is applied to each receptive field image. The"two-dimensional gas filter processing+polar transformation" isequivalent to the process of "one-dimensional filter processing afterpolar transformation", the simulation result is the same as FIGS.72A˜74. This system is the same as the stereopsis in the visual cortexof the cerebrum, and the contour-emphasized result of thetwo-dimensional gas filter can be also utilized for recognition of ashape, eyeball control, and the like.

3.6.1.3 Binocular Stereopsis of Edge

3.6.1.3.1 Binocular Stereopsis of Edge by One-Dimensional Filter

When replacing the process of "polar transformation+one-dimensionalfilter processing" according to the basic block diagram of FIG. 75, withthe process of "receptive field division+polartransformation+one-dimensional gradient filter+one-dimensional Gaussianfilter processing" indicated in FIG. 52B, and inputting image s of theright and left eyes, the binocular stereopsis of edges becomes possible.The simulation result of the above process is indicated in FIGS. 94A,94B, 95A and 95B. The circles portions C_(L) and C_(R) in FIGS. 94A and94B indicate receptive field images by the right and left eyes, whereina figure of edges in a portion encircled by a rectangular SQ is a sceneof an industrial plant. In FIGS. 95A and 95B, signal intensity of alldata is indicated by contour lines, the results of the polartransformation and the intensity distribution of the basic correlationparameter C(ρ,θ,σ) at θ=175° are indicated by contour lines. The sharppeak of C(ρ,θ=175°, σ) is a point corresponding to an edge, and precisemeasurement that the parallax is equal to one pixel, is performed basedon the values on the σ-axis.

3.6.1.3.2 Binocular Stereopsis of Edge wherein Two-Dimensional

Filter is Mixed (Binocular Stereopsis in the Visual Cortex of theCerebrum)

When replacing the process of "polar transformation+one-dimensionalfilter processing" according to the basic block diagram of FIG. 75, withthe process of "two-dimensional Gaussian filter+receptive fielddivision+polar transformation+one-dimensional gradient filterprocessing" indicated in FIG. 52C, and inputting image s of the rightand left eyes, the binocular stereopsis of edges becomes possible. Inthis system, two-dimensional second differentiation processing(two-dimensional convolution filter processing) is applied to the inputimages of the right and left eyes by the two-dimensional gas filter.Next, the filtered input images are respectively divided into receptivefield images, and polar transformation is applied to each receptivefield image. The "two-dimensional gas filter processing+polartransformation" is equivalent to the process of "one-dimensional gasfilter processing after polar transformation", the simulation result isthe same as FIGS. 94A and 94B. This system is the same as the stereopsisin the visual cortex of the cerebrum, and the contour-emphasized resultof the two-dimensional gas filter can be also utilized for recognitionof a shape, eyeball control, and the like.

3.6.1.4 Binocular Stereopsis of Polygon Figure, Curve Figure,Random-Dot, and Texture

Since the binocular stereopsis is related to the explanations ofmeasurement of movement direction and velocity, given below, thebinocular stereopsis of polygon figure, curve figure, random-dot, andtexture is explained in the sub-section 3.6.3 after the aboveexplanation of measurement of movement direction and velocity.

3.6.2 Measurement of Movement Direction and Movement Velocity

As a concrete example of images at different times, a system formeasuring the movement direction Φ and the velocity V of a moving objectfor pursuing the object, is explained.

3.6.2.1 Measurement of Movement Direction and Velocity of ContourTangential Line

The contour of an object is approximated as a straight line (tangentialline), the movement direction and the velocity of the tangential linecan be measured by the following method.

From data a_(t) (ρ,θ) of an image at the current time after theprocessing of "division into receptive fields+polartransformation+one-dimensional filter processing" is applied thereto,and data a_(t+DELTA) (ρ,θ) of an image at the next time after theprocessing of "polar transformation+one-dimensional filter processing"is applied thereto, the basic correlation parameter C(ρ,θ,τ) is given bythe following equation.

    C(ρ,θ,τ)=a.sub.t (ρ,θ)·a.sub.t+DELTA (ρ+τ,θ)                                     (124)

Then, a point (ρ_(P),θ_(P),τ_(P)) where C(ρ,θ,τ) is a local maximum isobtained. From the element parameters,

    movement direction of tangential line Φ=θ.sub.P +90° movement velocity of tangential line V=(τ.sub.P ·Δ.sub.ρ)/DELTA                        (125)

are obtained, and the tangential line can be pursued based on the data,where Δ.sub.ρ is a resolution in the ρ direction and DELTA is the delaytime, as indicated by reference numeral 349 in FIG. 77.

Errors are generated in the orientation and velocity of a tangentialline when the orientation and the movement direction V₀ of thetangential line do not cross at a right angle. The reason is explainedwith reference to FIG. 96. The movement direction and the velocity of atangential line L, measured according to the equation (125), are"direction perpendicular to the orientation of the tangential line" and"velocity V in the perpendicular direction", respectively, and thereforethe direction of the velocity V does not accord with the true directionvector (V₀,Φ₀). Although the method for measuring the correct movementdirection and the correct velocity, generated by correcting the errors,is explained next, the system is enough to pursue an object, since thepursuit is performed at a short time interval, and therefore the errorin the above equation at each pursuit is small enough.

FIG. 97 is a block diagram of a construction for measuring a movementdirection Φ and a movement velocity V of a tangential line. Image datais delayed by DELTA (349), image data of one frame at the current time,and image data of one frame and predetermined time before, obtained bythe delay, are respectively divided into receptive field images (350a,350a'), polar transformation processing is applied to each receptivefield image IM and IM' (351a, 351a'), one-dimensional filteringprocessing is applied to the result of the polar transformation to maponto dual planes (352a, 352a'), correlation processing is performedbetween data mapped onto the respective dual planes according to theequation (124) (353), the calculated correlation parameter C(ρ,θ,τ) isstored in the correlation parameter storage portion (372), after thecorrelation calculation is completed, the correlation parameter storageportion is scanned to detect (ρ_(P),θ_(P),τ_(P)) at which C(ρ,θ,τ) is alocal maximum (373), finally, the movement direction Φ and the movementvelocity V of the tangential line are calculated according to theequation (125) (374).

3.6.2.2 Measurement of Movement Direction and Velocity of Corner

For precisely measuring the movement direction Φ₀ and velocity V₀ of theobject, more than one contour line is necessary in a receptive field. Asindicated in FIG. 98A, generally, an object has a corner CN constitutedby two contour tangential lines L_(i) and L_(j), and therefore precisemeasurement of the movement direction Φ₀ and the velocity V₀ is possibleby the correlation processing of the corner.

In the same way as the case of sub-section 3.6.2.1, correlationprocessing is performed according to the equation (124). Namely,tangential line can be measured by the following method.

From data a_(t) (ρ,θ) of an image at the current time after theprocessing of "polar transformation+one-dimensional filter processing"is applied thereto, and data a_(t+DELTA) (ρ,θ) of an image at the nexttime after the processing of "polar transformation+one-dimensionalfilter processing" is applied thereto, the basic correlation parameterC(ρ,θ,τ) is calculated according to the equation (124), and a point(ρ_(P),θ_(P),τ_(P)) at which C(ρ,θ,τ) is a local maximum is obtained.Next, local maximum points (ρ_(i),θ_(i),τ_(i)) and (ρ_(j),θ_(j),τ_(j)),corresponding to two tangential lines L_(i) and L_(j) extracted by theabove method of local maximum value search, are obtained, and precisemovement of the direction Φ₀ and the velocity V₀ are given by thefollowing equations

    Φ.sub.0 =arctan[(τ.sub.i sin θ.sub.j -τ.sub.j sin θ.sub.i)/(τ.sub.i cos θ.sub.j -τ.sub.j cos θ.sub.i)],                                          (126)

and

    V=(Δ.sub.ρ ·τ.sub.i)/(sin (Φ-θ.sub.i)·DELTA)                     (127)

where DELTA is a time interval between the current time and the nexttime.

The equations (126) and (127) are derived as below. Namely, thedirection Φ and the velocity V of a tangential line L, measuredaccording to the equation (125) (see FIG. 96) are "directionperpendicular to the orientation of the tangential line L" and "velocityV in the perpendicular direction", respectively. Therefore, when movedin the direction Φ₀ of FIG. 96, the velocity measured by the equation(125) is equal to V₀ cosξ. Thus,

    V.sub.0 cos ξ=(τ.sub.P ·Δ.sub.ρ)/DELTA,(128)

where the angle ξ is given by the following equation, when the clockwisedirection is positive in FIG. 96,

    ξ=90°-(Φ.sub.0 -θ.sub.P)               (129)

where θ_(P) indicates the orientation of the tangential line L. Bysubstituting the equation (129) into the equation (128),

    V.sub.0 cos[90°-(Φ.sub.0 -θ.sub.P)]=(τ.sub.P ·Δ.sub.ρ)/DELTA                        (130)

is obtained, and the equation (129) is deformed to

    V.sub.0 sin (Φ.sub.0 -θ.sub.P)=(τ.sub.P ·Δ.sub.ρ)/DELTA.                       (130)'

Then, V₀ is obtained by putting τ_(P) =τ_(i) and θ_(P) =θ_(i) to derivethe equation (127). Further, when replacing τ_(P) with τ, and θ_(P) withθ, and substituting k=DELTA/Δρ into the equation (130), the followingequation

    τ=k·V.sub.0 ·cos [90°-(Φ.sub.0 -θ)](131)

is obtained. When drawing the above relationship of the equation (131)on the θ-τ plane, a sine wave (sinusoidal excitation pattern) isobtained as indicated in FIG. 98B. Namely, each side L_(i) and L_(j) ofa figure is transformed into one point on a sine wave on the θ-τ plane.

Deforming the equation (131),

    τ=k.sub.0 ·V.sub.0 ·sin (Φ.sub.0 -θ)(132)

is obtained. When substituting (θ,τ)=(θ_(i),τ_(i)) and(θ,τ)=(θ_(j),τ_(j)) into the equation (132), the following equations

    τ.sub.i =k·V.sub.0 ·sin (Φ.sub.0 -θ.sub.i),(132)'

and

    τ.sub.j =k·V.sub.0 ·sin (Φ.sub.0 -θ.sub.j(132)"

are obtained. From the equations (132)' and (132)",

    τ.sub.i /τ.sub.j =sin (Φ.sub.0 -θ.sub.i)/sin (Φ.sub.0 -θ.sub.j)

is obtained, and

    tan Φ.sub.0 (τ.sub.i sin θ.sub.j -τ.sub.j sin θ.sub.i)/(τ.sub.i cos θ.sub.j -τ.sub.j cos θ.sub.i)

is derived. That is, the equation (126) is derived.

Although a corner is constituted by two contour lines, a sine wave ofFIG. 98B can be determined more precisely when a characteristic featureis constituted by more than two contour lines, and therefore accuracy of(Φ₀,V₀) can be improved.

3.6.2.3 Measurement of Movement Direction and Velocity of Polygon andCurve

Although the measurement of movement direction and movement velocityfrom two lines (corner) is explained, further credible measurement ofmovement direction and movement velocity is possible when taking intoconsideration to "polygon and curve" which are constituted by many linesand tangential lines. The aforementioned correlation parameter C_(PRJ-)ρ(θ,τ), which is generated by the projection in the ρ direction, andgiven by the following equation, plays an important role.

    C.sub.PRJ-ρ (θ,τ)=ΣC(ρ,θ,τ)=(ρ=1, 2, . . . )

3.6.2.3.1 Measurement of Movement Direction and Velocity by Polygon

A parameter response to a polygon figure is, as indicated in FIG. 99,responses to respective sides of the polygon distributes in a sinusoidalform in the θ-τ plane. In the case of an N-sided polygon, N peaks lineup in the form of a sine wave. A polygon moving with a large velocityprovides a sine wave having a large amplitude. Thus, the movementdirection and the velocity of the whole polygon can be measuredprecisely.

When an angle made by a normal to a straight line and the true movementdirection is denoted by ξ as indicated in FIG. 96, τ_(P) at which thecorrelation parameter C(ρ,θ,τ) has a peak, is given from the equation(128) as below.

    τ.sub.P =(V.sub.0 ·DELTA/Δρ)·cos ξ

C_(PRJ-)ρ (θ,τ) generated by projecting the correlation parameterC(ρ,θ,τ) in the ρ direction, gives a sine wave (sinusoidal excitationpattern) of FIG. 99. By obtaining the maximum point (θ_(max),τ_(max)),

    movement direction Φ.sub.0 =θ.sub.max -90°,(133a)

and

    true movement velocity V.sub.0 =(τ.sub.max ·Δ.sub.ρ)/DELTA                        (133b)

are obtained. Comparing this method with the aforementioned cornermethod (FIG. 98B), the basic principles are the same. However, sincemany points (N points) are distributed on a sine wave, this method has aremarkable characteristic feature that the peak (a point at which thesine wave has a maximum amplitude) of the sine wave can be calculatedprecisely.

The extraction of the sine wave can be performed by using polartransformation. Namely, according to the polar transformation on acylinder (Hough transformation), points are transformed to sine waves.Inversely, when inverse polar transformation is applied to each point onthe sine wave (such inverse polar transformation is equivalent to theinversion Hough transformation), the respective points are transformed(inverse polar transformation) to a group of straight lines intersectingat a point. Therefore, from the point, the sine wave can be extracted.Concretely, respective points on the C_(PRJ-)ρ (θ,τ)-plane aretransformed to straight lines satisfying the following relationship

    τ=V.sub.y ·cos θ+V.sub.x ·sin θ.(134)

When the intersecting point CP(V_(x),V_(y)) of the straight lines isobtained (see FIGS. 100A and 100B), the direction and the distance fromthe origin of the V_(x) -V_(y) coordinate system to the intersectingpoint CP give Φ₀ and V₀ in the equations (133a) and (133b). The reasonwhy the direction and the distance from the origin of the V_(x) -V_(y)coordinate system to the intersecting point CP give Φ₀ and V₀ in theequation (133a) and (133b), is as follows.

When deforming the equation (134) with the true velocity V₀ anddirection Φ₀, ##EQU25## where Φ₀ =arctan(V_(y) /V_(x)).

Therefore, the peak of the sine wave on the C_(PRJ-)ρ (θ,τ)-plane isgiven by ##EQU26## Therefore, the true velocity V₀ and direction Φ₀ arecalculated by the following equations ##EQU27## This method correspondsto the "inversion Hough transformation".

FIG. 101 is a block diagram illustrating the construction for themeasurement of a movement direction and a movement velocity. Image dataof one frame at each of the current time and the next time is dividedinto receptive field images (350a, 350a'), polar transformationprocessing is applied to each receptive field image IM, IM' (351a,351a'), one-dimensional filtering processing is applied to the result ofthe polar transformation to map onto dual planes (352a, 352a'),correlation processing is performed between data mapped onto therespective dual planes according to the equation (124) (3539'), and thecalculated correlation parameter C(ρ,θ,τ) is stored in the correlationparameter storage portion (472). Then, the correlation parameterC_(PRJ-)ρ (θ,τ), projected in the ρ direction, is calculated by thefollowing equation (375). ##EQU28## The calculated correlation parameterC_(PRJ-)ρ (θ,τ) is stored in the θ-τ plane the correlation parameterstorage portion (376). Then, polar transformation processing is appliedto the correlation parameter C_(PRJ-)ρ (θ,τ) according to the equation(134) (377), and the peak point (intersecting point) on the V_(x) -V_(y)plane (velocity plane) is obtained (378). The direction and the distancefrom the origin of the coordinate system to the intersecting point areobtained, and the true velocity V₀ and the true direction Φ₀ arecalculated based on the equations (135a) and (135b) (379).

3.6.2.3.2 Measurement of Movement of Direction and Velocity from CurveFigure

Although the above explanations are given for a polygon, the abovemeasurement can be performed from a curve figure in a similar way.According to the process of "receptive field method+polartransformation", tangential lines of the curve can be preciselyextracted, and C_(PRJ-)ρ (θ,τ) can be calculated from the data of thetangential lines in the same way as above.

3.6.2.4 Measurement of movement of Direction and Velocity fromRandom-Dot and Texture

In the above explanations, it is described that the movement directionand the velocity can be measured from a figure (a polygon and a curve)which are constituted by straight lines and tangential lines. Next, thefigure (a polygon and a curve) can be extended to a figure constitutedby random points. From such a figure, the movement direction Φ₀ and thevelocity V₀ can be measured in the same processing as that in thesub-section 3.6.2.3. The reason is because, according to the process of"receptive field method+one-dimensional filter", "a pair of points" canbe extracted "as a straight line", and therefore the measurement fromthe figure constituted by random points can be performed in the same wayas the measurement from the polygon (see FIGS. 102A and 102B). Accordingto this method, a random-dot figure is naturally constituted by finedesign patterns, and therefore the movement direction and the velocityof a "texture" figure can be measured. This extension has a greatadvantage.

FIGS. 103A, 103B, 103C 103D, 104A and 104B are diagrams for explaining asimulation result of the measurement of the movement direction and thevelocity of a random-dot figure, and indicate a result in the case wherean object moves by 6√ 2 pixels per second in the 45° direction, wherethe delay DELTA is set equal to one second. The random-dot figure ofFIGS. 103A, 103B, 103C, 103D, 104A and 104B is a random-dot stereogramgenerated by a computer, where 1 dot=1 pixel, and the density is 50%.

In FIGS. 103A, 103B, 103C and 103D, IM and IM' are receptive fieldimages at the current time (without delay) and the next time (withdelay), respectively, and HCIM and HCIM' are hypercolumn images obtainedby performing polar transformation on the respective receptive fieldimages IM and IM', applying one-dimensional filtering processing to thepolar transformation results, and mapping the filtered result on to theρ-θ dual plane. In FIGS. 104A and 104B, PRIM indicates a correlationparameter C_(PRJ-)ρ (θ,τ) obtained by performing correlation processingin accordance with the equation (104) between the hypercolumn images,and projecting a correlation parameter C(ρ,θ,τ) obtained by thecorrelation processing, in the ρ direction onto the θ-τ plane; and HGIMindicates an inverse polar transformation result on the V_(x) -V_(y)plane, obtained by applying inverse polar transformation processing inaccordance with equation (134) to the correlation parameter C_(PRJ-)ρ(θ,τ). A sinusoidal excitation pattern SWV appears on the θ-τ plane, anda sharp peak PK is extracted at 45° direction and 6√ 2 pixels locationon the velocity plane. A true velocity V₀ and a movement direction Φ₀can be calculated from the peak point (V_(x),V_(y)) based on theequations (135a) and (135b).

Cells called "directional selective simple cells" or "directionalselective complex cells" exist in the hypercolumn in the primary visualcortex of the cerebrum, where the cells detect a moving direction and amoving velocity of a contour line, and the visual cortex (MT area)contains cells having a function of synthesizing information from thecomplex cells to detect the true velocity and the direction. Thefunction of the cells resembles the system according to the presentinvention. In particular, the fifth layer of the hypercolumn containscells which feedback signals to the superior colliculus, and the cellsplay an important role to control eyeballs so as to capture the objectat the center of the field of sight by measurement data of the truemovement direction Φ₀ and the moving velocity V₀ of the contour line.Since the operation of capturing an object at the center of the field ofsight to access or avoid the object, is required for moving robot andthe like, the method of measurement according to the present inventionis effective.

Recently, studies have been made regarding measurement methods of themoving direction Φ₀ and the moving velocity V₀. The studied methods aredivided into "methods for obtaining a moving direction and a movingvelocity based on a time variation of brightness" and "methods forobtaining a moving direction and a moving velocity based on a variationaccompanied by movement of the characteristic features of a figure".However, the former method has a problem in practice in that the methodis sensitive to variation and vibration of lighting due to use ofdifferentiation in the processing and there is a limit in application ofthe latter method to image processing techniques since current imageprocessing techniques are weak in "processing of a figure".

Regarding the above optical flow problem, the method of measurementaccording to the present invention is advantageous, compared with theformer method, in that the method of measurement according to thepresent invention is strong against noise since an integral operation isused in the method, in the manner that a group of points constituting afigure are transformed to straight lines by polar transformation, andaccumulation is performed. The method of measurement according to thepresent invention is advantageous, compared with the latter method, inthat measurement of a complicated figure can be performed accurately,since the complicated figure is decomposed to the simplest figures,i.e., contour tangential lines, and the contour tangential lines can beprocessed in the one-dimensional space due to polar transformation.Namely, the method of measurement according to the present invention canbe a new optical flow method which is an integral type and in whichone-dimensional processing is possible.

3.6.2.5 Measurement of Movement of Direction and Velocity of Line andGap

The processing for measuring a movement direction and a velocity of aline and a gap is explained below with a simulation result. Although, inthe following explanation, the method of 3.6.2.1 is used as a concretemeasurement method of a movement direction and a velocity, the methodsof 3.6.2.2˜3.6.2.4 can be performed as well. 3.6.2.5.1 Measurement ofMovement Direction and Velocity Using One-Dimensional Filter

The measurement of a moving direction Φ and a velocity V of a line and agap becomes possible by replacing the processing of "receptive fielddivision+polar transformation+one-dimensional filter processing" of FIG.97, with the processing of "receptive field division+polartransformation+one-dimensional Gaussian filter (or one-dimensionalGaussian filter+polarity inversion) processing" indicated in FIGS. 50Band 57B, respectively. The simulation result of the operation isindicated in FIGS. 105A to 105E. In FIGS. 105A to 105E, an intensity ofa correlation parameter C(ρ,θ,τ) obtained by inputting an original imageand an image obtained by shifting a perpendicular line (θ≈92°) by sevenpixels in the horizontal direction, is indicated by contour lines. Thesharp peak of the basic correlation parameter C(ρ,θ=92°,σ) indicates apoint indicating correspondence between the perpendicular lines. Whenthe point is denoted by (ρ_(P),θ_(P),τ_(P)) , the correct result isobtained as

    the movement direction=θ.sub.P +90°=182°,

and

    (movement velocity/DELTA)=τ.sub.P =6 image.

3.6.2.5.2 Measurement (Similar to the Visual Cortex of the Cerebrum) ofMovement Direction and Velocity Using Two-Dimensional Filter

The measurement of a movement direction and a velocity of a line and agap becomes possible by replacing the processing of "receptive fielddivision+polar transformation+one-dimensional filter processing" of FIG.97, with the processing of "two dimensional Gaussian filter +receptivefield division+polar transformation (or polar transformation+polarityinversion) processing" indicated in FIGS. 50C and 57C, respectively. Inthis processing, two-dimensional second differentiation processing(two-dimensional convolution filter processing) is performed on an inputimage with a two-dimensional gas filter, then the processed image isdivided into receptive fields, and polar transformation is performed oneach receptive field image. The processing of the "two-dimensional gasfilter+polar transformation" is equivalent to the processing of"one-dimensional gas filter after polar transformation". Although thesimulation result is not shown, the same measurement result of themovement direction and the velocity as FIGS. 105A to 105E is obtained.This processing is the same as the stereopsis in the visual cortex ofthe cerebrum, and it is advantageous that the contour-emphasized resultby the two-dimensional gas filter can be commonly used for recognitionof a figure, eyeball control, and the like.

3.6.2.6 Measurement of Movement Direction and Velocity of Edge

The processing for measuring a direction Φ and a velocity V of an edgeis explained below with a simulation result. The edge is the mostfrequently appeared image characteristic feature. Although, in thefollowing explanation, the method of 3.6.2.1 is used as a concretemeasurement method of a movement direction and a velocity, the methodsof 3.6.2.1˜4 can also be performed as well.

3.6.2.6.1 Measurement of Movement Direction and Velocity of Edge UsingOne-Dimensional Filter

The measurement of a moving direction and a velocity of an edge becomespossible by replacing the processing of "receptive field division+polartransformation+one-dimensional filter processing" of FIG. 97, with theprocessing of "receptive field division+polartransformation+one-dimensional Gaussian filter (+polarity inversion)processing" indicated in FIG. 52B. The simulation result of theoperation is indicated in FIGS. 106A to 106E. In FIG. 106E, an intensityof a correlation parameter C(ρ,θ,σ) obtained by inputting an originalimage and an image obtained by shifting an edge at the direction of 120°by seven pixels in the horizontal direction, is indicated by contourlines. The sharp peak of the basic correlation parameter C(ρ,θ=120°,σ)indicates a point indicating correspondence between the edges. When thepoint is denoted by (ρ_(P),θ_(P),τ_(P)) , the correct result is obtainedas

    the movement direction=θ.sub.P +90°=210°,

and

    (movement velocity·DELTA)/Δ.sub.ρ =τ.sub.P =7 image,

where Δ.sub.ρ is a resolution of ρ direction, and DELTA is a delay time.

3.6.2.6.2 Measurement of Movement Direction and Velocity of Edge UsingTwo-Dimensional Filter (Detection of Movement in Visual Cortex ofCerebrum)

The measurement of a moving direction Φ and a velocity V of an edgebecomes possible by replacing the processing of "receptive fielddivision+polar transformation+one-dimensional filter processing" of FIG.97, with the processing of "two-dimensional Gaussian filter+receptivefield division+polar transformation+one-dimensional gradient filterprocessing" indicated in FIG. 52C. The processing of the two-dimensionalgas filter+polar transformation is equivalent to the processing of"one-dimensional gas filter after polar transformation". Although thesimulation result is not shown, the same measurement result of themovement direction and the velocity as FIG. 106E is obtained. Thisprocessing is the same as the detection of movement in the visual cortexof the cerebrum, and it is advantageous that the contour-emphasizedresult by the two-dimensional gas filter can be commonly used forrecognition of a figure, eyeball control, and the like.

3.6.3 Binocular Stereopsis of Polygon Figure, Curve Figure, andRandom-Dot Texture

3.6.3.1 Binocular Stereopsis of Polygon Figure and Curve Figure

Similar to sub-section 3.6.2 "Measurement of Movement Direction andVelocity", the binocular stereopsis of a polygon figure and a curvefigure, becomes possible by using a projected correlation parameterC_(PRJ-)ρ (θ,ρ) which is obtained by projecting a correlation parameterC(ρ,θ,σ) in the ρ-direction, as ##EQU29## This processing can beperformed by replacing the time correlation C_(PRJ-)ρ (θ,τ) insub-section 3.6.2 with the two-eye correlation C_(PRJ-)ρ (θ,σ).According to this method, credibility is improved because a polygoncontains a plurality of sides.

A straight line is seen in input images of the right and left eyes,offset from each other as indicated by SL_(L) and SL_(R) in FIG. 107.The difference in the locations of the straight line location isobtained as a displacement (minimum distance) σ in the ρ-directionperpendicular to the orientation θ of the straight line, on the ρ-θhypercolumn plane. Therefore, the binocular parallax d in the horizontaldirection (minimum distance between SL_(L) and SL_(R) is expressed bythe equation

    d=σ/ sin θ.

Namely, when the horizontal parallax (binocular parallax) is denoted byd, the displacement σ in the θ-direction of the straight line isexpressed by the following equation

    σ=d·sin θ.                            (136)

When drawing the relationship of the above equation on the θ-σ plane, asine wave as indicated in FIG. 108B is obtained. Namely, each sideL_(i), L_(j), and L_(k) in each of the images captured by right and lefteyes as indicated in FIG. 108A, are polar-transformed, correlationprocessing is performed between hypercolumn images obtained by the polartransformation based on the equation (101) to calculate a correlationparameter C(ρ,θ,σ), the correlation parameter C(ρ,θ,σ) is projected inthe ρ direction according to the equation (109) to obtain a correlationparameter C_(PRJ-)ρ (θ,σ) and display on the θ-σ plane. Each side L_(i),L_(j), and L_(k) are transformed to points L_(i) ', L_(j) ', and L_(k) 'on the sine wave.

Similar to the processing in sub-section 3.6.2.3, when C_(PRJ-)ρ (θ,σ)is calculated from the two eyes input images, andinverse-polar-transformed according to the equation (134), respectivepoints on the sine wave are transformed to a group of straight linespassing through a point PK on the V_(x) -V_(y) plane as indicated inFIG. 108C. When this peak point PK(V_(x),V_(y)) is obtained, the trueparallax σ0 is calculated by the equation ##EQU30## where the directionof the parallax is always the same as the direction (horizontaldirection) connecting two eyes.

When a direction in which an arbitrary figure is seen is denoted by ξ,as indicated in FIG. 109, and the space between two eyes is denoted byd, the distance D to the arbitrary figure is calculated by the equation

    D=d·sin (σ.sub.0 +ξ)/ sin (σ.sub.0)(138)

FIG. 109 is a diagram for explaining a method of calculating a distanceto an arbitrary figure by the binocular stereopsis, where L is astraight line, and E_(L) and E_(R) denote right and left eyes. When theabove obtained parallax is denoted by σ₀, the direction in which thearbitrary figure is seen is denoted by ξ, the space between two eyes isdenoted by d, distances from the right and left eyes to the arbitraryfigure are denoted by D₁ and D₂, respectively, the equations

    D.sub.1 sin ξ=D.sub.2 sin (ξ+σ.sub.0)

    D.sub.1 cos ξ-D.sub.2 cos (ξ+σ.sub.0)=d

are obtained. By obtaining D₁ from these simultaneous equations, theequation (138) is obtained.

FIG. 110 is a block diagram for measurement of a parallax and a distanceto an arbitrary figure by the binocular stereopsis. Image data by eachof the right and left eyes is divided into receptive field images (350a,350a'), each receptive field image IM and IM' is polar-transformed(351a, 351a'), one-dimensional filtering processing is applied to thepolar transformation result to map on a dual plane (352a, 352a'),correlation processing of the equation (101) is performed between datamapped on the respective dual planes (653), and the calculatedcorrelation parameter C(ρ,θ,σ) is stored in the correlation parameterstorage portion (572). Next, a correlation parameter C_(PRJ-)ρ (θ,σ) isobtained by projection in the ρ-direction according to the equation(109) (375') to be stored in the correlation parameter storage portionof the θ-σ plane (376'). Then, inverse polar transformation processingis performed on the correlation parameter C_(PRJ-)ρ (θ,σ) according tothe equation (134) (377'), a peak point (intersecting point) on theV_(x) -V_(y) plane is obtained (378'), a distance from the coordinateorigin to the intersecting point is obtained, and the binocular parallaxσ₀ is obtained based on the equation (137), and the distance D to thearbitrary figure is calculated based on the equation (138) (379').

3.6.3.2 Binocular Stereopsis of Random-Dot Texture

By the same method as 3.6.3.1, binocular stereopsis of "a random-dotfigure and a texture figure" becomes possible. Thereby, binocularstereopsis of a plane containing a fine design pattern can be performedeasily. This is a model at a level of cell, of a fact proved first by apsychologist, Julesz, that "even a random-dot figure which itself cannotbe recognized as a figure, can be recognized by the binocularstereopsis".

FIGS. 111A to 111D and 112A to 112C are diagrams for explaining asimulation result of binocular stereopsis of a random-dot figure, in thecase where random-dot figures different (displaced in the horizontaldirection) by six pixels from each other are input into right and lefteyes, respectively. These random-dot figures are random-dot stereographgenerated by a computer, where 1 dot=1 pixel, and the density is 50%. InFIGS. 111A to 111D, 1M and 1M' respectively denote receptive fieldimages of the right and left eyes, and HCIM and HCIM' respectivelydenote hypercolumn images generated by applying polar transformation tothe receptive field images IM and IM', applying one-dimensionalfiltering processing to the polar transformation result to map onto theρ-θ dual planes. In FIGS. 112A to 112C, PRIM denotes a correlationparameter C_(PRJ-)ρ (θ,σ) on the θ-σ plane, obtained by performingcorrelation processing according to the equation (101), betweenhypercolumn images, and projecting the correlation parameter C(ρ,θ,σ) inthe ρ direction according to the equation (109), HGIM denotes a resultof inverse polar transformation applied to the correlation parameterC_(PRJ-)ρ (θ,σ) on the V_(x) -V_(y) plane according to the equation(134), generated by applying polar transformation processing tocorrelation parameter C_(PRJ-)ρ (θ,σ). A sine wave pattern SWV appearson the θ-σ plane, and a sharp peak PK is extracted at a location apartfrom the origin by six pixels in the horizontal direction on the V_(x)-V_(y) plane. The binocular parallax σ₀ and a distance D to thearbitrary figure is calculated from the peak point (V_(x),V_(y)) basedon the equations (137) and (138).

3.6.4 Kinetic Stereopsis

3.6.4.1 Kinetic Stereopsis of Straight Line

In sub-section 3.6.2, the method for obtaining a velocity and a movementdirection of a moving object is explained. Using the same processing, adistance (depth) to a straight line in a space is measured by moving animage capture means (camera or the like). Namely, by obtainingcorrelation parameter C(ρ,θ,τ) based on the equation (104)', and a peakpoint (ρ_(P),θ_(P),τ_(P)) of the correlation parameter, a motionparallax between the image without delay and the image with delay, iscalculated as τ_(P) ·Δ₉₂ . When the direction in which the straight lineis seen is denoted by ξ, the movement velocity of the image capturemeans by V_(S). the delay in the equation (104)' by DELTA, the distanceto the straight line is calculated by the equation (138), by theequation

    D=(V.sub.S ·DELTA)·sin (τ.sub.P ·Δ.sub.ρ +ξ)/ sin (τ.sub.P ·Δ.sub.P).                                 (139)

FIG. 113 is a block diagram of measurement of a depth to a straight linein a space by kinetic stereopsis. The image before the movement isdelayed (349), each of the image without delay a_(t) +(ρ,θ) and theimage with delay a_(t) +DELTA(ρ+τ,θ) is divided into receptive fieldimages (350a, 350a'), polar transformation processing is applied to eachreceptive field image IM and Im' (351a, 351a'), one-dimensionalfiltering processing is applied to the polar transformation results tomap onto dual planes (352a, 352a'), correlation processing according tothe equation (104)' is performed between the data mapped on therespective dual planes (453), and the calculated correlation parameterC(ρ,θ,τ) is stored in the correlation parameter storage portion (372').Next, a peak point (ρ_(P),θ_(P),τ_(P)) of the correlation parameterC(ρ,θ,τ) is obtained (373'), and a depth to a straight line in a spaceis calculated using the parallax τ_(P) ·Δ.sub.ρ according to theequation (139) (391).

3.6.4.2 Kinetic Stereopsis of Arbitrary Figure

A distance (depth) to an arbitrary figure in a space is measured bymoving an image capture means (camera or the like). Namely, the sameprocessing as explained in sub-section 3.6.2.3 is performed on theimages before and after the movement of the image capture means todetermine a peak point on the V_(x) -V_(y) plane, and a true velocity V₀and a movement direction is calculated according to the equations (135a)and (135b). When the direction in which the figure is seen is denoted byξ, the movement velocity of the capture means by V_(S), and the delay inthe equation (104)' by DELTA, the distance to the arbitrary figure iscalculated, in the same manner as the equation (138), by the equation

    D=(V.sub.S ·DELTA)·sin (V.sub.0 ·DELTA+ξ)/ sin (V.sub.0 ·DELTA).                            (140)

FIG. 114 is a block diagram of measurement of a depth to an arbitraryfigure in a space by the kinetic stereopsis. The image is delayed (349),each of the image without delay a_(t) (ρ,θ) and the image with delaya_(t+DELTA) (ρ+τ,θ) is divided into receptive field images (350a,350a'), polar transformation processing is applied to each receptivefield image IM and IM' (351a, 351a'), one-dimensional filteringprocessing is applied to the polar transformation results to map ontodual planes (352a, 352a'), correlation processing according to theequation (104)' is performed between the data mapped on the respectivedual planes (753), and the calculated correlation parameter C(ρ,θ,τ) isstored in the correlation parameter storage portion (3721). Next, acorrelation parameter C_(PRJ-)ρ (θ,τ) which is projected in theρ-direction according to the equation (110) (3751) to store in thecorrelation parameter storing portion (3761), a peak point(ρ_(P),θ_(P),τ_(P)) of the correlation C(ρ,θ,τ) is obtained according tothe equation (110) (373'), and a depth to the arbitrary figure in aspace is calculated using the parallax τ_(P) ·Δ.sub.ρ according to theequation (140) (3991). Then, inverse polar transformation processing isapplied to the correlation parameter C_(PRJ-)ρ (θ,τ) according to theequation (134) (3771) to obtain a peak point (intersecting point) on theV_(x) -V_(y) plane (3781), and a true velocity V₀ and a true orientationΦ₀ are calculated according to the equations (135a) and (135b ) (3791).Finally, a depth to an arbitrary figure is calculated according to theequation (140) (3991).

3.6.5 Generalization

The correlation processing system according to the present invention isgenerated as indicated in FIG. 115. Namely, polar transformationprocessing is applied to input data to map onto the ρ-θ dual plane(3501), processing (1) of adding a new parameter to thepolar-transformed data a(ρ,θ), correlation processing (2), or projectionwith an arbitrary parameter or polar transformation processing (3)(projection in the direction of 0 or 90° is equivalent to polartransformation) is applied, thereby data a(ξ₁,ξ₂, . . . ) including asinusoidal excitation pattern on a plane, or data including anexcitation pattern of a great circle on a sphere is obtained (3502), andthe sinusoidal excitation pattern or the excitation pattern of the greatcircle included in the data, is extracted by inverse polartransformation processing to output useful data (3503).

In the equations (101) and (104), the "correlation processing" isperformed with introducing a new parameter a or τ. In the equations(109) and (110), the "projection along the parameter ρ" in performed.Further, in the equation (119), projection in the 45° direction on theρσ plane is performed, where the projection is equivalent to theoperation of "polar transformation of the (ρ,σ) plane" and extraction ofthe components in the 45° direction.

In the measurement of the movement direction and the velocity explainedin sub-sections 3.6.2.2 and 3.6.2.3, polar transformation processing isapplied to data of two images at different time to obtain data a(ρ,θ)and b(ρ,θ) in the ρ-θ dual plane (3501), correlation processing isperformed according to the equation (104) introducing the ρ-directionmovement velocity parameter τ, and projecting the correlation parameterC(ρ,θ,τ) in the ρ direction according to the equation (110) to obtainthe projection data C_(PRJ-)ρ (θ,τ) including the sinusoidal excitationpattern (see FIGS. 98A, 98B and 99) (3502). Then, inverse polartransformation processing is applied to the projection data C_(PRJ-)ρ(θ,τ) to extract a sinusoidal excitation pattern (giving a local maximumpoint) and output useful data (movement direction and velocity) (3503).

The "inverse polar transformation" extracting the sinusoidal excitationpattern is expressed as ##EQU31## In the above equation, δ() denotes adelta function, and τ_(X) and τ_(Y) denote velocity parameters in the Xand Y-axis directions. Since the delta function δ() is equal to one atthe point of zero, and zero at the other points, the above equation isdeformed to ##EQU32## which makes clear the content of the inverse polartransformation. This calculation is carried out in the embodiment.

In the measurement by the binocular stereopsis explained in sub-section3.6.3, polar transformation processing is applied to the data of twoimages captured by the two eyes to obtain data L(ρ,θ) and R(ρ,θ) on theρ-θ dual plane (3501). Next, correlation processing is performedaccording to the equation (101) introducing the parallax σ in theρ-direction, the correlation parameter C(ρ,θ,σ) is projected in the ρdirection according to the equation (109) to obtain the projection dataC_(PRJ-)ρ (θ,σ) including the sinusoidal excitation pattern (see FIGS.108A, 108B and 108C) (3502). Then, inverse polar transformationprocessing is applied to the projection data C_(PRJ-)ρ (θ,σ) to extractthe sinusoidal excitation pattern (giving the local maximum point) andoutput useful data (parallax) (3503).

The "inverse polar transformation" extracting the sinusoidal excitationpattern is expressed as ##EQU33## where δ() denotes a delta function,and σ_(X) and σ_(Y) denote parameters in the X and Y-axis directions.Since the delta function δ() is equal to one at the point of zero, andzero at the other points, the above equation is deformed to ##EQU34##

This calculation is carried out in the embodiment.

Although, in the above processing, the sinusoidal excitation pattern isobtained by performing the correlation processing and the like in step3502 in FIG. 115, the sinusoidal excitation pattern can be obtained bypolar transformation without correlation processing and the like. In thelatter case, in step 3502, transformation processing such as a shift anda rotation is applied to the (ρ,θ) data obtained by the polartransformation in step 3501, and inverse polar transformation processingof step 3503 is applied to the (ρ,θ) data obtained by the transformationprocessing to extract the sinusoidal excitation pattern (giving thelocal maximum point) and output useful data. For example, when applyingpolar transformation to the circle of the radius R indicated in FIG.116A, a sine wave, which is shifted by R in the ρ direction is obtainedas indicated in FIG. 116B. When the center of the circle in denoted by(x₀,y₀), ##EQU35## are obtained. Therefore, by shifting the dataobtained by the polar transformation, by R in the ρ direction, andapplying inverse polar transformation processing to data a(θ,σ) obtainedby the shift processing to extract a sinusoidal excitation pattern(giving a local maximum point), a circle center can be output (3503).

FIGS. 117A to 117D are diagrams for explaining of a simulation result ofthe circle detection. In the simulation, Laplacian filter processing isapplied to a black circle drawn on a white paper to obtain input imagedata, the input image is divided into receptive field images, polartransformation processing is applied to the receptive field image toobtain (ρ,θ) data containing the sinusoidal excitation pattern on theρ-θ plane, then the (ρ,θ) data is shifted by R in the ρ direction, andinverse polar transformation is applied to the data obtained by theshift processing. In FIGS. 117A to 117D, intensities of the excitationare indicated by contour lines, and positive portions are indicated byhatching. In FIG. 117B, two parallel sine waves SN₁ and SN₂ are obtainedby the polar transformation, since the range of ρ-2ρ in the equation(141) is inverted to the negative side of ρ based on the considerationof the physiological knowledge on the hypercolumn, that "the orientationθ of the straight line stimulus detected by the hypercolumn is limitedto the range of 0˜ρ". As understood from FIG. 117A, one sine wave isexcited as a sharp "negative peak PK" corresponding to the center of thecircle, and the other sine wave is excited as the weak ring RG of aradius 2R.

3.7 Advantage of Third Aspect of Present Invention

Since, according to the third aspect of the present invention, polartransformation processing is applied to input data to project onto adual plane, or further filter processing is applied to the polartransformation processing result to project onto the dual plane, thencorrelation processing is performed between mapped data on the dualplane to measure a variable specifying relationship between figurativecharacteristic features (for example, tangential lines) of the inputdata, corresponding portions in a plurality of figures can be determinedsimply, and precisely with a small amount of processing, and thus thefunctions of the binocular stereopsis (measurement of a binocularparallax, a depth, measurement of optical flows, a movement velocity anda movement direction of an object, and the like) can be realized.

Since polar transformation processing, filter processing, andcorrelation processing are performed on each receptive field imagegenerated by dividing one frame into receptive fields which are smallareas, the amount of processing can be greatly reduced.

When the receptive field images belong to different images taken by twoor three cameras, the functions of the binocular stereopsis and thestereopsis by three eyes can be realized. In addition, when thereceptive field images belong to images at different times, the movementdirection and the movement velocity of characteristic features (lines,corners, and the like) captured in the receptive fields, can bemeasured, and therefore movement while capturing an object at the centerof the field of view, becomes possible, and application to movablerobots, unmanned vehicles, and the like becomes possible. Further, whenthe receptive field images belong to images before and after themovement of an image capture means such as a camera, the function of thekinetic stereopsis can be realized and a depth to an object can bemeasured.

When the receptive field images belong to images in different receptivefields of the same image (screen), or images in the same receptive fieldimage, texture analysis whereby a degree of the same design pattern inan image can be analyzed.

When correlation processing among a plurality of receptive field imagesis performed for each color, for each color difference signal, or foreach primary color, and therefore, the binocular stereopsis, the pursuitof a movement object, and texture analysis can be performed moreprecisely.

When one-dimensional Gaussian filter processing is performed after polartransformation, or two-dimensional Gaussian filter processing isperformed before polar transformation processing, corresponding linesand gaps in a plurality of figures can be obtained. When one-dimensionalgradient filter processing and one-dimensional Gaussian filterprocessing are performed after polar transformation, or two-dimensionalgradient filter processing before polar transformation processing andone-dimensional Gaussian filter processing after polar transformationare performed, corresponding edges in a plurality of figures can beobtained. Further, a location, an orientation, a parallax, a movementdirection, and a movement velocity of the above figure elements can beobtained.

When a plurality of receptive field images belong to spatially differentimages, and the correlation parameter Cθ(ρ,θ,σ) in the θ-direction iscalculated, or correlation parameter C(ρ,θ,σ₁,σ₂) in the (ρ,θ) plane iscalculated, extraction of a moving tangential line changing itsorientation becomes possible.

When a plurality of receptive field images belong to images at differenttimes, and the correlation parameter C(ρ,θ,τ) in the ρ-direction iscalculated, a location, an orientation, and a velocity of a translatingtangential line can be quantitatively obtained. Further, the correlationparameter C(ρ,θ,τ₁,τ₂) in the (ρ,θ) plane is calculated, a location, anorientation, a movement velocity, and a rotation velocity of atangential line moving with changing the orientation can bequantitatively obtained.

When the correlation parameter C(ρ,θ,σ) is projected in the σ-axisdirection indicating a spatial shift amount σ, in the ρ-directionindicating a tangential line location, in the θ-direction indicating atangential line orientation, or in an arbitrary two-axis direction, thememory capacity storing correlation parameters can be reduced by anamount corresponding to one or two axes. Further, when a projectiondirection is selected, a desired value of a location, a parallax, anorientation, and the like of a tangential line, can be obtained.

When the correlation parameter C(ρ,θ,σ) is projected in the τ-axisdirection indicating a time shift amount (movement velocity) τ, in theρ-direction indicating a tangential line location, in the θ-directionindicating a tangential line orientation, or in an arbitrary two-axisdirection, the memory capacity storing correlation parameters can bereduced by an amount corresponding to one or two axes. Further, when aprojection direction is selected, a desired value of a location, anorientation, a translation velocity, a rotation velocity, and the likecan be obtained.

When polar transformation processing is performed on receptive fieldimages to project onto the ρ-θ dual plane, picking up a combination ofmapped data a(ρ,θ) in the dual plane and mapped data b(ρ,θ) in anotherdual plane, where the coordinate values of the data differ by apredetermined amount in the ρ-direction, and a sum of products iscalculated, precise filtering can be performed, and it is preferable toapply the above processing to the "extraction of a characteristicfeature seen as the same by the right and left eyes" and the "pursuit ofthe same characteristic feature as the previous image".

When shifting, with regard to mapped data a(ρ,θ) in the dual plane,another mapped data b(ρ,θ) in the ρ-direction or in the θ-direction,subtraction from the mapped data a(ρ,θ) is performed, and then thesubtraction is repeated with varying the shift amount to supply theobtained subtraction result as the correlation parameter, the binocularstereopsis and the pursuit of an object the contour of which is obscure,based on a gradual variation of brightness or hue.

                  TABLE 1                                                         ______________________________________                                        Relationship between                                                          Lens and Projection Surface                                                             Projection                                                                             Wideness   Resolution                                                Surface  of Sight   of Angle                                        ______________________________________                                        Standard/   plane      X narrow   O high                                      Telephoto                                                                     Lens                                                                          Cylindrical cylindrical                                                                              D angular  D axis                                      Lens                   direction  direction                                                          only       only                                        Fisheye     sphere     O wide     X low                                       Lens                                                                          ______________________________________                                    

                  TABLE 2                                                         ______________________________________                                        Extraction of Line Segment                                                    from Standard/Telephoto Lens                                                  ______________________________________                                        (Part 1)                                                                      (a-1) Polar Transformation on Plane                                           (processing on plane)                                                         • processing is simple because of "transformation                        between straight line and point"                                             • processing is simplest due to drawing of straight                      line.                                                                        (Vicinity of origin corresponds to infinity, and                              inversion transformation in (a-4) is more practical.)                         (a-2) Polar Transformation on Cylinder                                        (processing on plane)                                                         • Developing on a plane is possible, and                                 "transformation between sine wave and point".                                • Processing is second simplest due to drawing of sine                   wave.                                                                        • Same transformation as "Hough transformation"                          algebraically derived from parameter space method,                            and being considered as geometrical version of the                            parameter space method.                                                      •Utilization in extraction of line segment from plane                    image                                                                        ______________________________________                                        (Part 2)                                                                      (a-3) Polar Transformation on Sphere                                          (processing on sphere)                                                        • "transformation between great circle and point"                       • process of drawing of great circle                                    • Suitable but wide field of view of spherical                           projection is mostly not utilized due to narrow                               field of view of Standard/Telephoto Lens.                                    • Equivalent to processing of (a-2) when range of                        sight is narrow.                                                             (a-4) Synthetic Inversion Transformation on Plane                             (processing on plane)                                                         • "transformation between point and circle. passing                      through origin"; inverse transformation of (a-1)                             • characteristic features                                               a. Real image and dual image are drawn in same plane                          b. Dual image (hypercolumn) is representation by                                polar coordinate                                                            c. Process of drawing circle is necessary                                     ______________________________________                                    

                  TABLE 3                                                         ______________________________________                                        Extraction of Line Segment                                                    from Cylindrical Lens                                                         ______________________________________                                        (b-1) Polar Transformation on Plane                                           (processing on plane)                                                         • "transformation between straight line and point"                      • Not practical since wide field of view in angular                     direction of cylindrical input is damaged.                                    (b-2) Polar Transformation on Cylinder                                        (processing on plane)                                                         • Developing on plane is possible, and "transformation                  between sine wave and point".                                                 • Processing is relatively simple due to drawing of                      sine wave (Hough transformation)                                             • most suitable for cylindrical lens.                                   Note: Hough transformation is provided for                                    cylindrical lens.                                                             (b-3) Polar Transformation on Sphere                                          (processing on sphere)                                                        • "transformation between great circle and point"                       • Process of drawing of great circle is necessary.                      • Suitable but wide field of view of spherical                           projection is not mostly utilized due to narrow                               field of view of cylindrical lens in axis direction                           of cylinder.                                                                 ______________________________________                                    

                  TABLE 4                                                         ______________________________________                                        Extraction of Line Segment                                                    from Fisheye Lens                                                             ______________________________________                                        (c-1) Polar Transformation on Plane                                           (processing on plane)                                                         • "transformation between straight line and point"                      • Not practicai since wide field of view in sphere                       input is damaged.                                                            (c-2) Polar Transformation on Cylinder                                        (processing on plane)                                                         • Developing on a plane is possible, and                                 "transformation between sine wave and point".                                • Not practical since wide fieid of view in sphere                       input is damaged.                                                            (c-3) Polar Transformation on Sphere                                          (processing on sphere)                                                        • "transformation between great circle and point"                       • Processing of drawing of great circle is necessary.                   • most suitable for fisheye lens                                        (c-4) Synthetic Inversion Transformation on Plane                             (processing on plane)                                                         • "transformation between point and circle passing                      through fixed point on the sphere"; inversion                                 transformation of (c-3)                                                       • characteristic features                                               i. Real image and dual image are drawn in                                       same sphere.                                                                ii. Dual image (hypercolumn) is representation                                  by polar coordinate.                                                        • Process of drawing of great circle is necessary.                      ______________________________________                                    

                  TABLE 5                                                         ______________________________________                                        Suitability of Polar                                                          Transformation Surface with Lens                                                        Surface for Polar                                                             Transformation and                                                            Transformation                                                                            Suitable Lens                                           ______________________________________                                        Polar       processing on standard/telephoto                                  Transformation                                                                            plane:        lens                                                on Plane    straight line                                                                 ←→ point                                              Polar       processing on cylindrical lens,                                   Transformation                                                                            plane:        standard/telephoto                                  on Cyiinder sine wave     lens, and fisheye                                               ←→ point                                                                        lens                                                Polar       processing on fisheye lens,                                       Transformation                                                                            sphere:       cylindrical lens, and                               on Sphere   great circle  standard/telephoto                                              ←→ point                                                                        lens                                                Polar       Arbitrary     lens of same shape as                               Transformation                                                                            surface:      Polar Transformation                                on Arbitrary                                                                              straight line surface                                             Surface     in broad sense                                                                ←→ point                                                                        underlined portion:                                                           optimum lens                                        ______________________________________                                    

                  TABLE 6                                                         ______________________________________                                        Synthetic Evaluation of                                                       Various Types of Processing                                                   Type  Processing Time                                                                              Hardware Size  Index of                                  (FIG. Processing         Hardware       Synthetic                             No.)  Time      Ratio    Size    Ratio  Evaluation                            ______________________________________                                        31A   (m + m)N.sup.2                                                                          2        mN.sup.2                                                                              1      2                                     31B   (m.sup.2 + m)N.sup.2                                                                    m + 1    m.sup.2 N.sup.2                                                                       m      m(m + 1)                              31C   mN.sup.2  1        (m + m)N.sup.2                                                                        2      1                                     31D   m.sup.2 N.sup.2                                                                         m        (m.sup.2 + m)N.sup.2                                                                  m + 1  m(m + 1)                              ______________________________________                                         (Note)                                                                        Index of Synthetic Evaluation = (Processing Time) × (Hardware Size)

I claim:
 1. An image processing process comprising:a first step fordividing an original image into small areas; a second step for applyingpolar transformation to the original image in each of the small areas,to obtain a polar-transformed image of the original image in each of thesmall areas, where the polar transformation is a transformation betweenan n-dimensional hyperplane containing an origin in an (n+1)-dimensionalAffine space, and a vector passing through the origin and perpendicularto the n-dimensional hyperplane, at least one image element of each ofthe original images in each of the small areas corresponds to one ofsaid n-dimensional hyperplane and said vector, and a polar-transformedimage element which is polar-transformed from each of said at least oneimage element corresponds to the other of said n-dimensional hyperplaneand said vector; and a third step for applying one-dimensional filteringon the polar-transformed image.
 2. An image processing process accordingto claim 1, wherein the one-dimensional filtering is one-dimensionalfirst differential filtering.
 3. An image processing process accordingto claim 2, further comprising a fifth step for extracting a figureelement corresponding to an edge existing in each of the small areas ofthe original image, based on the result of the processing ofone-dimensional first differential filtering.
 4. An image processingprocess according to claim 1, wherein, in the third step, a plurality ofone-dimensional first differential filtering are applied at the sametime to the result of the polar transformation, wherein the plurality ofone-dimensional first differential filtering have differentcharacteristics, respectively, andthe fourth step comprises a sub-stepfor synthesizing results of the plurality of one-dimensional firstdifferential filtering.
 5. An image processing process according toclaim 1, wherein the one-dimensional filtering is one-dimensionalfiltering by a skeleton filter the characteristics of which can beexpressed by a δ-function.
 6. An image processing process comprising:afirst step for dividing an original image into small areas; a secondstep for applying polar transformation to the original image in each ofthe small areas, to obtain a polar-transformation image of the originalimage in each of the small areas, where the polar transformation is atransformation between an n-dimensional hyperplane containing an originin an (n+1)-dimensional Affine space, and a vector passing through theorigin and perpendicular to the n-dimensional hyperplane, at least oneimage element of each of the original images in each of the small areascorresponds to one of said n-dimensional hyperplane and said vector, anda polar-transformed image element which is polar-transformed from eachof said at least one image element corresponds to the other of saidn-dimensional hyperplane and said vector; a third step for applyingone-dimensional second differential filtering on the solar-transformedimage; and a fourth step for extracting a figure element correspondingto a line existing in each of the small areas of the original image fromthe result of the one-dimensional second differential filtering.
 7. Animage processing process according to claim 6, wherein, in the thirdstep, a plurality of one-dimensional second differential filtering areapplied at the same time to the result of the polar transformation,wherein the plurality of one-dimensional second differential filteringhave different widths, respectively, andthe fourth step comprises asub-step for synthesizing the results of the plurality ofone-dimensional second differential filtering.
 8. An image processingprocess according to claim 6 further comprising another step after thethird step for dividing the results of the plurality of one-dimensionalsecond differential filtering into positive signals and negativesignals, respectively; andthe fourth step comprises:a first sub-step forsynthesizing the positive signals obtained from the pluralityone-dimensional second differential filtering to output the synthesizedresult, and a second sub-step for synthesizing the negative signalsobtained from the plurality of one-dimensional second differentialfiltering to output the synthesized result.
 9. An image processingprocess comprising:a first step for dividing an original image intosmall areas; a second step for applying polar transformation to theoriginal image in each of the small areas, to obtain a polar-transformedimage of the original image in each of the small areas, where the solartransformation is a transformation between an n-dimensional hyperplanecontaining an origin in an (n+1)=dimensional Affine space, and a vectorpassing through the origin and perpendicular to the n-dimensionalhyperplane, at least one image element of each of the original images ineach of the small areas corresponds to one of said n-dimensionalhyperplane and said vector, and a polar-transformed image element whichis polar-transformed from each of said at least one image elementcorresponds to the other of said n-dimensional hyperplane and saidvector; a third step for applying one-dimensional second differentialfiltering on the polar-transformed image; a fourth step for invertingthe polarity of the result of the one-dimensional second differentialfiltering; and a fifth step for extracting a figure elementcorresponding to a gap existing in each of the small areas of theoriginal image from the result of the inverting.
 10. An image processingprocess according to claim 9, wherein the third step comprises:a firstsub-step for applying a plurality of one-dimensional second differentialfiltering at the same time to the result of the polar transformation,wherein the plurality of types of one-dimensional second differentialfiltering have different widths, respectively, and a second sub-step forsynthesizing the results of the plurality of first differentialfiltering.
 11. An image processing process according to claim 9, furthercomprising:another step before the third step for applying a pluralityof one-dimensional first differential filtering at the same time to theresult of the polar transformation, wherein the plurality ofone-dimensional first differential filtering have different widths,respectively; and an additional step for synthesizing the results of theplurality of first differential filtering.
 12. An image processingprocess comprising:a first step for dividing an original image intosmall areas; a second step for applying polar transformation to theoriginal image in each of the small areas, to obtain a polar-transformedimage of the original image in each of the small areas, where the polartransformation is a transformation between an n-dimensional hyperplanecontaining an origin in an (n+1)-dimensional Affine space, and a vectorpassing through the origin and perpendicular to the n-dimensionalhyperplane, at least one image element of each of the original images ineach of the small areas corresponds to one of said n-dimensionalhyperplane and said vector, and a polar-transformed image element whichis polar-transformed from each of said at least one image elementcorresponds to the other of said n-dimensional hyperplane and saidvector; a third step for applying one-dimensional first differentialfiltering on the polar-transformed image; and a fourth step forextracting a figure element corresponding to an edge existing in each ofthe small areas of the original image from the result of the third step.13. An image processing process comprising:a first step for dividing anoriginal image into small areas; a second step for applying polartransformation to the original image in each of the small areas, toobtain a polar-transformed image of the original image in each of thesmall areas, where the polar transformation is a transformation betweenan n-dimensional hyperplane containing an origin in an (n+1)-dimensionalAffine space, and a vector passing through the origin and perpendicularto the n-dimensional hyperplane, at least one image element of each ofthe original images in each of the small areas corresponds to one ofsaid n-dimensional hyperplane and said vector, and a polar-transformedimage element which is polar-transformed from each of said at least oneimage element corresponds to the other of said n-dimensional hyperplaneand said vector; a third step for applying one-dimensional firstdifferential filtering on the polar-transformed image; a fourth step forapplying one-dimensional second differential filtering on the result ofthe one-dimensional first differential filtering; and a fifth step forextracting a figure element corresponding to an edge existing in each ofthe small areas of the original image from the result of the fourthstep.
 14. An image processing process according to claim 13, furthercomprising an additional step, before the fourth step, for separatingthe result of the one-dimensional differential filtering into positiveand negative signals; andthe fourth step comprises:a first sub-step forapplying one-dimensional second differential filtering on the positiveand negative signals, respectively, and a second sub-step forsynthesizing the results of one-dimensional second differentialfiltering obtained from the positive and negative signals, foroutputting the synthesized result.
 15. An image processing processaccording to claim 13, wherein, in the third step, a plurality ofone-dimensional first differential filtering are applied at the sametime to the result of the polar transformation, where the plurality oftypes of one-dimensional first differential filtering have differentwidths, respectively, andin the fourth step, one-dimensional seconddifferential filtering is applied on the results of the plurality oftypes of one-dimensional first differential filtering to obtain aplurality of results of the one-dimensional second differentialfiltering, and the fifth step comprises a sub-step for synthesizing theresults of the plurality of one-dimensional second differentialfiltering.
 16. An image processing process according to claim 15 furthercomprising another step for separating the results of the plurality ofone-dimensional differential filtering into positive signals andnegative signals, respectively;in the fourth step, one-dimensionalsecond differential filtering is applied to the positive signals andnegative signals obtained from the plurality of one-dimensionaldifferential filtering to output a plurality of positive signals and aplurality of negative signals, and the another step comprises:a firstsub-step for synthesizing the plurality of positive signals to outputthe synthesized result, and a second sub-step for synthesizing theplurality of negative signals to output the synthesized result.
 17. Animage processing process comprising:a first step for dividing anoriginal image into small areas; a second step for applying polartransformation to the original image in each of the small areas, toobtain a polar-transformed image of the original image in each of thesmall areas, where the polar transformation is a transformation betweenan n-dimensional hyperplane containing an origin in an (n+1)-dimensionalAffine space, and a vector passing through the origin and perpendicularto the n-dimensional hyperplane, at least one image element of each ofthe original images in each of the small areas corresponds to one ofsaid n-dimensional hyperplane and said vector, and a polar-transformedimage element which is polar-transformed from each of said at least oneimage element corresponds to the other of said n-dimensional hyperplaneand said vector; a third step for applying one-dimensional seconddifferential filtering on the polar-transformation image; a fourth stepfor extracting a figure element corresponding to a line existing in eachof the small areas of the original image based on the result of thethird step and for producing a fourth step result; a fifth step forapplying one-dimensional first differential filtering on thepolar-transformed image and for producing a fifth step result; and asixth step for extracting a line which does not coincide with the lineobtained in the fourth step, from the fifth step result, as a figureelement corresponding to an edge existing in each of the small areas ofthe original image.
 18. An image processing process comprising:a firststep for dividing an original image into small areas; a second step forapplying polar transformation to the original image in each of the smallareas, to obtain a polar-transformed image of the original image in eachof the small areas, where the polar transformation is a transformationbetween an n-dimensional hyperplane containing an origin in an(n+1)-dimensional Affine space, and a vector passing through the originand perpendicular to the n-dimensional hyperplane, at least one imageelement of each of the original images in each of the small areascorresponds to one of said n-dimensional hyperplane and said vector, anda polar-transformed image element which is polar-transformed from eachof said at least one image element corresponds to the other of saidn-dimensional hyperplane and said vector; a third step for applyingone-dimensional second differential filtering on the polar-transformedimage; a fourth step for extracting a candidate of a figure elementcorresponding to a line existing in each of the small areas of theoriginal image from the result of the third step; a fifth step forapplying one-dimensional first differential filtering on thepolar-transformed image; a sixth step for extracting a candidate of afigure element corresponding to an edge existing in each of the smallareas of the original image from the result of the fourth step; and aseventh step for extracting figure elements corresponding to a line andan edge, respectively, existing in each of the small areas of theoriginal image from the candidates of the figure elements correspondingto the line and the edge of the fourth and sixth steps.
 19. An imageprocessing process comprising:a first step for dividing an originalimage into small areas; a second step for applying polar transformationto each pixel in the original image in each of the small areas, andobtaining a curve on a predetermined dual plane as a polar-transformedimage element of each pixel in the original image in each of the smallareas, where the polar transformation is a transformation between ann-dimensional hyperplane containing an origin in an (n+1)-dimensionalAffine space, and a vector passing through the origin and perpendicularto the n-dimensional hyperplane, each pixel of the original image ineach of the small areas corresponds to said vector, and said curve assaid polar-transformed image element which is polar-transformed fromeach pixel corresponds to said n-dimensional hyperplane; a third stepfor obtaining an accumulated polar-transformed image on the dual planeby accumulatively storing in a memory having a storage area for eachpixel on the dual plane, a value of each pixel in the original image, asa value of each pixel constituting said curve on the dual plane; and afourth step for applying one-dimensional filtering on the accumulatedpolar-transformed image in each of the small areas.
 20. An imageprocessing process according to claim 19, wherein, in the fifth step, apoint at which a peak of the values accumulatively stored in the memoryis located in the polar-transformed image on the dual plane, isrecognized as a line existing in each of the small areas of the originalimage.
 21. An image processing apparatus comprising:a first means fordividing an original image into small areas; a second means for applyingpolar transformation to the original image in each of the small areas,to obtain a polar-transformed image of the original image in each of thesmall areas, where the polar transformation is a transformation betweenan n-dimensional hyperplane containing an origin in an (n+1)-dimensionalAffine space, and a vector passing through the origin and perpendicularto the n-dimensional hyperplane, at least one image element of each ofthe original images in each of the small areas corresponds to one ofsaid n-dimensional hyperplane and said vector, and a polar-transformedimage element which is polar-transformed from each of said at least oneimage element corresponds to the other of said n-dimensional hyperplaneand said vector; and a one-dimensional filter for applyingone-dimensional filtering on the polar-transforming image.
 22. An imageprocessing apparatus comprising:first means for dividing an originalimage into small areas; second means for applying polar transformationto each pixel in the original image in each of the small areas, andobtaining a curve on a predetermined dual plane as a polar-transformedimage element of each pixel in the original image in each of the smallareas, where the polar transformation is a transformation between ann-dimensional hyperplane containing an origin in an (n+1)-dimensionalAffine space, and a vector passing through the origin and perpendicularto the n-dimensional hyperplane, each pixel of the original image ineach of the small areas corresponds to said vector, and said curve assaid polar-transformed image element which is polar-transformed fromeach pixel corresponds to said n-dimensional hyperplane; a memory havinga storage area for each pixel on the dual plane; third means forobtaining an accumulated polar-transformed image on the dual plane byaccumulatively storing in the memory, a value of each pixel in theoriginal image, as a value of each pixel constituting said curve on thedual plane; and a one-dimensional filter for applying one-dimensionalfiltering on the accumulated polar-transformed image.
 23. A computerreadable product storing at least one program which, when executed by aprocessor, causes the processor and its associated hardware to carry outthe steps of:a first step for dividing an original image into smallareas; a second step for applying polar transformation to the originalimage in each of the small areas, to obtain a solar-transformed image ofthe original image in each of the small areas, where the polartransformation is a transformation between an n-dimensional hyperplanecontaining an origin in an (n+1)-dimensional Affine space, and a vectorpassing through the origin and perpendicular to the n-dimensionalhyperplane, at least one image element of each of the original images ineach of the small areas corresponds to one of said n-dimensionalhyperplane and said vector, and a polar-transformed image element whichis polar-transformed from each of said at least one image elementcorresponds to the other of said n-dimensional hyperplane and saidvector; and a third step for applying one-dimensional filtering on thepolar-transformed image.
 24. A program executing machine containing acomputer readable product storing at least one program which, whenexecuted by the program executing machine, causes the program executingmachine and its associated hardware to carry out the steps of:a firststep for dividing an original image into small areas; a second step forapplying polar transformation to the original image in each of the smallareas, to obtain a polar-transformed image of the original image in eachof the small areas, where the polar transformation is a transformationbetween an n-dimensional hyperplane containing an origin in an(n+1)-dimensional Affine space, and a vector passing through the originand perpendicular to the n-dimensional hyperplane, at least one imageelement of each of the original images in each of the small areascorresponds to one of said n-dimensional hyperplane and said vector, anda polar-transformed image element which is polar-transformed from eachof said at least one image element corresponds to the other of saidn-dimensional hyperplane and said vector; and; a third step for applyingone-dimensional filtering on the polar-transformed image.
 25. An imageprocessing process comprising:dividing an original image into smallareas; applying polar transformation to the original image in each ofthe small areas, wherein said polar transformation comprises ann-dimensional hyperplane containing an origin in an (n+1)-dimensionalaffine space and a vector passing through the origin and perpendicularto the n-dimensional hyperplane; and applying one-dimensional filteringto the result of the polar transformation.